Parameter-Efficient Fine-Tuning for Pre-Trained Vision Models: A Survey

Large-scale pre-trained vision models (PVMs) have As a promising solution, parameter-efficient fine-tuning shown great potential for adaptability across various (PEFT), which was originally proposed in NLP, overcomes downstream vision tasks. However, with stateof-the-art the above challenges by updating a minimal number of parameters PVMs growing to billions or even trillions while potentially achieving comparable or superior of parameters, the standard full fine-tuning performance to full fine-tuning [Hu and et al., 2021; Yu and paradigm is becoming unsustainable due to high et al., 2022]. These approaches hinge on recent advances computational and storage demands. In response, showing that large pre-trained models trained with rich data researchers are exploring parameter-efficient finetuning have strong generalisability and most parameters in the PVMs (PEFT), which seeks to exceed the performance could be shared for the new tasks [Kornblith and et al., 2019; of full fine-tuning with minimal parameter Yu and et al., 2022]. PEFT methods could reduce learnable parameters, modifications. This survey provides a comprehensive which not only facilitates more effective adaptation overview and future directions for visual PEFT, to novel tasks but also safeguards the pre-existing knowledge offering a systematic review of the latest advancements.