A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendations

Abstract--Modern deep neural networks, particularly recent large language models, come with massive model sizes that require significant computational and storage resources. To enable the deployment of modern models on resource-constrained environments and accelerate inference time, researchers have increasingly explored pruning techniques as a popular research direction in neural network compression. More than a thousand pruning papers have been published each year from 2020 to 2022. However, there is a dearth of up-to-date comprehensive review papers on pruning. To address this issue, in this survey, we provide a comprehensive review of existing research works on deep neural network pruning in a taxonomy of 1) universal/specific speedup, 2) when to prune, 3) how to prune, and 4) fusion of pruning and other compression techniques. We then provide a thorough comparative analysis of seven pairs of contrast settings for pruning (e.g., unstructured/structured, one-shot/iterative, data-free/data-driven, initialized/pretrained weights, etc.) and explore several emerging topics, including post-training pruning, different levels of supervision for pruning to shed light on the commonalities and differences of existing methods and lay the foundation for further method development. Finally, we provide some valuable recommendations on selecting pruning methods and prospect several promising research directions for neural network pruning. To facilitate future research on deep neural network pruning, we summarize broad pruning applications (e.g., adversarial robustness, natural language understanding, etc.) and build a curated collection of datasets, networks, and evaluations on different applications. We will keep updating this repository to include the latest advancements in the field. Over the past several years, Deep Neural Networks resources (such as CPU, GPU, and memory), energy, and (DNNs) have achieved conspicuous progress in various bandwidth [11, 12, 13]. Although DNNs achieve including fast real-time response and compact memory remarkable success in various areas, their performance footprint. Deep neural networks' computational complexity heavily relies on model parameters and computational cost. With the popularity of large 95MB memory for storage, contains over 23 million trainable language models in recent years, there is growing interest parameters, and requires 4 GFLOPs (Giga Floating Point in compressing neural networks for computers with flexible Operations) of computations [7]. In addition, deep neural networks trained on ImageNet [1] is more than 500 MB [8]. The that contain redundant features can undermine their Transformer network GPT-3 model consists of up to 175 robustness, elevating the risk of adversarial attacks [16].