CrossPT: Exploring Cross-Task Transferability through Multi-Task Prompt Tuning

Modern NLP systems increasingly rely on parameter-efficient adaptation methods to customize large pre-trained language models (PLMs) for new tasks without updating all model parameters. Among these, prompt tuning has become popular for its simplicity and low memory footprint: it learns a small set of continuous prompt embeddings while keeping the PLM frozen [5]. This allows adaptation to many tasks with minimal added parameters for both language models [2] and vision-language models [4]. Despite this efficiency, most existing prompt tuning methods are designed for single-task learning, where prompts are optimized independently for each task without sharing knowledge across them. This design is limiting in multi-task settings, where tasks often share semantic structure, labels, or data domains that can be exploited for transfer. Learning isolated prompts from scratch for each task misses opportunities for cross-task knowledge sharing--especially problematic in few-shot scenarios where data is scarce. Moreover, with only a single prompt vector per task, it becomes difficult to represent complex task relationships or adapt to varying label spaces, risking both underfitting of shared patterns and overfitting of task-specific noise. These challenges motivate the need for a more modular and transferable approach to prompt tuning that can share generalizable representations across tasks while preserving task-specific specialization in a parameter-efficient manner.