Differentiable Hardware

How AI Might Help Revive the Virtuous Cycle of Moore's Law In the wake of the global chip shortage, TSMC has reportedly raised chip prices and delayed the 3nm process. Whether or not it is accurate or indicative of a long-term trend, this kind of news should alert us to the worsening impact of the decline of Moore's Law and compel a rethinking of AI hardware. Would AI hardware be subject to this decline or help reverse it? Suppose we want to revive the virtuous cycle of Moore's Law, in which software and hardware propelled one another, making a modern smartphone more capable than a past-decade warehouse-occupying supercomputer. A popularly accepted post-Moore virtuous cycle, in which bigger data leads to larger models requiring more powerful machines, is not sustainable. We can no longer count on transistor shrinking to build wider and wider parallel processors unless we redefine parallelism. Nor can we rely on Domain-Specific Architecture (DSA) unless it facilitates and adapts to software advancement.