Frankle et al.[11] proposed an iterative pruning with rewinding to avoid lower-stability phase to SGDnoiseintheearlytraining ineachIMPstep.

Work was done when the author interned at Microsoft.

Adepiction 4. The architecture 2. Ourf is

Writing testerror(test) w and w0 ( 1 )err(w0))].

As illustrated by the Lottery TicketHypothesis (LTH), pruning also has the potential of improving their generalization ability. At the core of LTH, iterative magnitude pruning (IMP) isthepredominant pruning method tosuccessfully find'winning tickets'. Yet, the computation cost of IMP grows prohibitively as the targeted pruning ratio increases.

A large body of research since the 1980s empirically observed that large neural networks can be compressed orsparsified toasmallfraction oftheiroriginal sizewhilemaintaining theirpredictive accuracy[14-16,20,23,29,45].