Improving Adaptability and Generalizability of Efficient Transfer Learning for Vision-Language Models