Customizable Combination of Parameter-Efficient Modules for Multi-Task Learning

Modular and composable transfer learning is an emerging direction in the field of Parameter Efficient Fine-Tuning, as it enables neural networks to better organize various aspects of knowledge, leading to improved cross-task generalization. In this paper, we introduce a novel approach Customized Polytropon (C-Poly) that combines task-common skills and task-specific skills, while the skill parameters being highly parameterized using low-rank techniques. Each task is associated with a customizable number of exclusive specialized skills and also benefits from skills shared with peer tasks. A skill assignment matrix is jointly learned. To evaluate our approach, we conducted extensive experiments on the Super-NaturalInstructions and the SuperGLUE benchmarks. Our findings demonstrate that C-Poly outperforms fully-shared, task-specific, and skill-indistinguishable baselines, significantly enhancing the sample efficiency in multi-task learning scenarios. As the number of parameters in Large Language Models (LLMs) continues to grow, training these models efficiently with limited computational resources has become a challenge. In recent years, there has been a shift towards employing Parameter Effective Fine-Tuning (PEFT) methods to address this issue. Examples of such methods include LoRA (Hu et al., 2022), AdaLoRA (Zhang et al., 2023a), and (IA) These methods focus on fine-tuning the adapter while freezing the pre-trained model, effectively reducing the computational cost.