A Multi-Level Superoptimizer for Tensor Programs

For We introduce Mirage, the first multi-level superoptimizer for a given algorithm, these optimizers automatically generate tensor programs. A key idea in Mirage is Graphs, a uniform performant kernels by searching over possible strategies for representation of tensor programs at the kernel, thread block, executing the kernel on the target hardware. However, due and thread levels of the GPU compute hierarchy. Graphs to the linear algebra nature of DNNs, a tensor program can enable Mirage to discover novel optimizations that combine be represented by a wide spectrum of mathematically equivalent algebraic transformations, schedule transformations, and algorithms, and existing schedule-based optimizers only generation of new custom kernels. To navigate the large consider kernels whose algorithms are manually specified search space, Mirage introduces a pruning technique based by users, resulting in missed optimization opportunities.