SARA: Singular-Value Based Adaptive Low-Rank Adaption

With the increasing number of parameters in large pre-trained models, LoRA as a parameterefficient fine-tuning(PEFT) method is widely used for not adding inference overhead. The LoRA method assumes that weight changes during fine-tuning can be approximated by lowrank matrices. However, the rank values need to be manually verified to match different downstream tasks, and they cannot accommodate the varying importance of different layers in the model. In this work, we first analyze the relationship between the performance of different layers and their ranks using SVD. Based on this, we design the Singular-Value Based Adaptive Low-Rank Adaption(SARA), which adaptively finds the rank during initialization by performing SVD on the pre-trained weights. Additionally, we explore the Mixture-of-SARA(Mo-Figure 1: An overview of our methods, (a) performing SARA), which significantly reduces the number SVD on the pre-trained weights and determining of parameters by fine-tuning only multiple the number k of values that account for a proportion parallel sets of singular values controlled by a threshold m of the total sum of singular values; (b) the router. Extensive experiments on various complex method of adding a truncated singular value matrix to tasks demonstrate the simplicity and parameter the pre-trained weights based on k; and (c) the extreme efficiency of our methods. They can method of fine-tuning only mixture of parallel singular effectively and adaptively find the most suitable values.