A Analysis

A.1 Analysis of Layerwise Pruning Strategies A.1.1 Fixed Pruning Rate We consider an L-layer neural network and each layer has n The following theorem shows, the fixed pruning rate strategy is the strategy which approximates the maximization of the total number of combinations. We defer the proof to Appendix B.1. Therefore, we can consider the fixed pruning rate strategy, i.e., 1 r = Drawbacks While this strategy may seem intuitive, it does not take the differences in layer size into account. In practice, the number of parameters in different layers can vary widely. For example, residual networks [30] have much fewer parameters in the first and last layers than in the middle ones. Using fixed pruning rate thus yields very few parameters after pruning within such small layers.