MLKD-BERT: Multi-level Knowledge Distillation for Pre-trained Language Models

MLKD-BERT conducts a (Liu et al., 2019). The PLM usually has large number two-stage distillation for feature-level as well as of parameters and long inference time, making relation-level knowledge, with 6 distillation loss it inapplicable to resource-limited devices and realtime functions designed for embedding layer, Transformer scenarios. Therefore, it is crucial to reduce layers, and prediction layer. Compared with PLM's storage and computation overhead while previous works, we have made two main contributions: retaining its performance. Knowledge distillation (Hinton et al., 2015) is an effective technique for PLM compression. In knowledge distillation, a In addition to feature-level knowledge, our smaller compact student model is trained, under student model learns valuable relation-level the guidance of a larger complicated teacher model, knowledge (relation among tokens and relation to keep similar model performance.