PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models

LoRA freezes the original model W and updates the "Noise & Zero" adapter, which may lead to slow convergence. To overcome this limitation, we introduce Principal Singular values and Singular vectors Adaptation (PiSSA).