Appendix for " Fine-Tuning Pre-Trained Language Models Effectively by Optimizing Subnetworks Adaptively "

In Sec.3.3, we have experimentally verified that DPS outperforms various fine-tuning methods. Table 1: Eight datasets used in this paper form GLUE benchmark. In this paper, we investigate the performance of DPS on five distinctive and widely used large-scale pre-trained language models, namely BERT Devlin et al. [2018], RoBERTa Liu et al. [2019], DeBERTa improves Transforme-based pre-trained model with disentangled attention mechanism and enhanced mask decoder. We use mixed precision training to speed up the experimental process. This method is applied by ELECTRA when fine-tuning downstream tasks. 2 D Appendix D. Experimental Details for Different Fine-tuning Methods The following is our hyperparameter search space for different fine-tuning regularization methods: Mixout We grid search Mixout probability p {0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8}.