Appendix A Derivation of (3) Based on the fact that the θ (m) is satisfied with the stationary condition of the lower-level objective function in (2), we obtain

Masks may vary between each iteration, and the pruned weights are indicated using the light gray color. Different colors of the edges in the neural networks refer to the weight update. The initial learning rate for all the methods are 0.1. All the evaluations are based on a single Tesla-V100 GPU. P do not require additional epochs for retraining.