A Survey on Transformer Compression

Abstract--Large models based on the Transformer architecture play increasingly vital roles in artificial intelligence, particularly within the realms of natural language processing (NLP) and computer vision (CV). Model compression methods reduce their memory and computational cost, which is a necessary step to implement the transformer models on practical devices. Given the unique architecture of transformer, featuring alternative attention and Feedforward Neural Network (FFN) modules, specific compression techniques are required. The efficiency of these compression methods is also paramount, as it is usually impractical to retrain large models on the entire training dataset. This survey provides a comprehensive review of recent compression methods, with a specific focus on their application to transformer models. The compression methods are primarily categorized into pruning, quantization, knowledge distillation, and efficient architecture design. In each category, we discuss compression methods for both CV and NLP tasks, highlighting common underlying principles. At last, we delve into the relation between various compression methods, and discuss the further directions in this domain. For example, When quantizing a full-precision model (MLP), convolutional neural network (CNN), recurrent neural (float32) into 8-bit integers, the memory cost can be reduced network (RNN), long short-term memory (LSTM), Transformers, by a factor of four. In recent times, transformer-based models have emerged as the be divided into post-training quantization(PTQ) or quantizationaware prevailing choice across various domains, including both natural training (QAT), in which the former only incurs limited language processing (NLP) and computer vision (CV) domains. Knowledge Considering their strong scaling ability, most of the large models distillation serves as a training strategy, which transfers knowledge with over billions of parameters are based on the transformer from a large model (teacher) to a smaller model (student). The architecture, which are considered as foundational elements for student mimics the behavior of the teacher by emulating the general artificial intelligence (AGI) [1], [2], [3], [4], [5], [6]. Notably, for advanced While large models have demonstrated significant capabilities, models like GPT-4, accessible only through APIs, their generated their exceptionally vast sizes pose challenges for practical instructions and explanations can also guide the learning of the development. For instance, the GPT-3 model has 175 billion student model [7], [8].In addition to obtaining models from predefined parameters and demands approximately about 350GB memory large models, some methods yield efficient architectures model storage (float16). The sheer volume of parameters and by directly reducing the computational complexity of attention the associated computational expenses necessitate devices with modules or FFN modules.