Zero-Shot Continuous Prompt Transfer: Generalizing Task Semantics Across Language Models

Recently in natural language processing (NLP), there has been a paradigm shift from full language model finetuning to the optimization of a small subset of prompt tokens (Shin et al., 2020; Lester et al., 2021; Li and Liang, 2021; Zhong et al., 2021). As language models have dramatically increased in size and may contain billions of parameters (Brown et al., 2020), the strategy of freezing language models while optimizing the learnable prompt parameters becomes the most affordable and efficient alternative for downstream tasks. This technique, referred to as prompt tuning, has gained substantial recognition for its efficacy across a range of language models (Shin et al., 2020; Lester et al., 2021; Li and Liang, 2021; Zhong et al., 2021). Various prompt tuning methods have been explored, which can be generally categorized into discrete and continuous cases. Discrete prompt tuning, such as AutoPrompt (Shin et al., 2020), primarily focuses on the selection and optimization of a predetermined set of tokens within a language model's vocabulary.