fdc42b6b0ee16a2f866281508ef56730-Supplemental.pdf

To estimate the impact of removing a parameter, these methods often use importance measures that were originally designed to prune neural networks. If this hypothesis is true, it has great potential to covert the inefficient training process on a large network to the scalable training process over a small one with comparable test accuracy. Most of existing LTH techniques provide empirical evidence to verify the LTH, although these methods raise very intriguing observations [71, 12, 1, 47, 69, 54, 5, 53, 26, 8, 7, 11]. However, multiple cycles of training and pruning over large neural networks are time-consuming. Tworecent worksanalyze the LTH transferability, i.e., the ticket discovered from one source task can be transferred to another targettask[44,43].