Block-Sparse GPU Kernels

The development of model architectures and algorithms in the field of deep learning is largely constrained by the availability of efficient GPU implementations of elementary operations. One issue has been the lack of an efficient GPU implementation for sparse linear operations, which we're now releasing, together with initial results using them to implement a number of sparsity patterns. These initial results are promising but not definitive, and we invite the community to join us in pushing the limits of the architectures these kernels unlock. Dense layers (left) can be replaced with layers that are sparse and wide (center) or sparse and deep (right) while approximately retaining computation time. Sparse weight matrices, as opposed to dense weight matrices, have a large number of entries with a value of exactly zero.