VB-LoRA: Extreme Parameter Efficient Fine-Tuning with Vector Banks