Results are shown inTable 3. Figure 1 and 2 illustrate the differences between 1:2 and 2:4 with the same dense weight matrix and sparsity (i.e. For every four elements, two will be removed based on the magnitude criterion. We first apply DominoSearch to search the layer-wise schemes from pre-trained dense models. For ResNet18/50 [4], the pre-trained models have been downloaded from Pytorch Model zoo1.

Furthermore, sparsity acceleration relies on the underlying software-architecture system of the platform. We can make a reasonable assumption that there will be more but not arbitrary N:M schemes supported infuture software-architecture designs.

Neural pruning is a widely-used compression technique for Deep Neural Networks (DNNs).

Section 2: Experimental study of a different policy with fixed N and flexible M. Section 3: Sensitivity of hyper-parameter β In the main paper, we assume a policy with fixed M and flexible N. Furthermore, we also use a design space with N equal to a power-of-two. This is achieved by transforming the schemes of fixed M. For instance, 8:16, 4:16, 2:16 and 1:16 will be transformed as 1:2, 1:4, 1:8 and 1:16 with fixed N (1) and flexible M (2,4,8,16). Results are shown in Table 3. Figure 1 and 2 illustrate the differences between 1:2 and 2:4 with the same dense weight matrix and sparsity (i.e. Details can be found in Section 3.4 of the main paper. It consists of more than 1.2 million training images and Each image is labelled as one of 1K classes.

Neural pruning is a widely-used compression technique for Deep Neural Networks (DNNs). However, the existing N:M algorithms only address the challenge of how to train N:M sparse neural networks in a uniform fashion (i.e. To tackle this problem, we present a novel technique -- \textbf{\textit{DominoSearch}} to find mixed N:M sparsity schemes from pre-trained dense deep neural networks to achieve higher accuracy than the uniform-sparsity scheme with equivalent complexity constraints (e.g. For instance, for the same model size with 2.1M parameters (87.5\% sparsity), our layer-wise N:M sparse ResNet18 outperforms its uniform counterpart by 2.1\% top-1 accuracy, on the large-scale ImageNet dataset. For the same computational complexity of 227M FLOPs, our layer-wise sparse ResNet18 outperforms the uniform one by 1.3\% top-1 accuracy.