Appendix

Supposeσ(θ,z) Rd,we define the data-dependent featureby φMatrix(θ)= h σ(θ,z(1)),...,σ(θ,z(m)) i> Rm d, Notice that the score of most neuron can be calculated cheaply usingqi,A(k) and gi,A(k). The only exception are neuron withai(k) > 0 and γi,A(k) < ai(k)/(1 ai(k)). Andthenwegenerate thetraining data by sampling featurex from Unif[0,1] (each coordinate is sampled independently) and then generatelabely = Fgen(x). Wealso include the pruned model using global imitation,whichisdenotedas Fnglobal. Following the layer-wise procedure introduced in Section 2.4, suppose that the algorithms prunes F` to f`,A`, ` [L].