Review for NeurIPS paper: MPNet: Masked and Permuted Pre-training for Language Understanding

Summary and Contributions: In this paper, the authors propose a new pre-trained language model called MPNet, which combines the advantages of both BERT (MLM) and XLNet (PLM). The proposed MPNet leverages the dependency among predicted tokens through PLM and takes auxiliary position information as input to reduce the position discrepancy. In practice, to combine MLM and PLM poses great challenge, and the authors propose several delicate designs to overcome the issues. The experiments are mainly carried out on a base model (i.e., 110M params), and the results show that the proposed MPNet could give consistent and significant improvements over similar baseline models. Overall, the idea of the paper is straightforward and easy to understand, which is a natural extension to combine the MLM and PLM.