Know Where You're Going: Meta-Learning for Parameter-Efficient Fine-Tuning

A recent family of techniques, dubbed lightweight fine-tuning methods, facilitates parameter-efficient transfer learning by updating only a small set of additional parameters while keeping the parameters of the pretrained language model frozen. While proven to be an effective method, there are no existing studies on if and how such knowledge of the downstream fine-tuning approach should affect the pretraining stage. In this work, we show that taking the ultimate choice of fine-tuning method into consideration boosts the performance of parameter-efficient fine-tuning. By relying on optimization-based meta-learning using MAML with certain modifications for our distinct purpose, we prime the pretrained Figure 1: Transfer learning for NLP pipeline; the model specifically for parameter-efficient finetuning, shaded block is our contribution. Conventional transfer resulting in gains of up to 1.7 points practice (dashed arrows) does not differentiate between on cross-lingual NER fine-tuning. Our ablation full fine-tuning and parameter-efficient fine-tuning in settings and analyses further reveal that any way. This work proposes a meta-learning solution the tweaks we introduce in MAML are crucial to further modify and prime a pretrained model parameters for the attained gains.