ESPN: Extremely Sparse Pruned Networks

Neural networks have achieved state of the art results across several domains such as computer vision, language processing, and reinforcement learning. This performance is generally contingent on large, over-parameterized networks that are trained using massive amounts of data. For example, the current state of the art on the ImageNet classification task uses a network with over 480 million parameters [TVDJ20], and consequently, the best performing networks are prohibitive in terms of computational and memory requirements. Therefore, compressing neural networks is vital for resource limited settings (such as self-driving cars and mobile devices). In this work, we present an algorithm for compressing neural networks to far higher degree of sparsity levels than has been reported in the literature.