Enabling Lightweight Fine-tuning for Pre-trained Language Model Compression based on Matrix Product Operators

MPO decomposition is structural in terms of information Recently, pre-trained language models (PLMs) (Devlin distribution: the central tensor with most et al., 2019; Peters et al., 2018; Radford et al., of the parameters encode the core information of 2018) have made significant progress in various the original matrix, while the auxiliary tensors with natural language processing tasks. Instead of training only a small proportion of parameters play the role a model from scratch, one can fine-tune a PLM of complementing the central tensor. Such a property to solve some specific task through the paradigm motivates us to investigate whether such an of "pre-training and fine-tuning". MPO can be applied to derive a better PLM compression Typically, PLMs are constructed with stacked approach: can we compress the central Transformer layers (Vaswani et al., 2017), involving tensor for parameter reduction and update the auxiliary a huge number of parameters to be learned.