Parameter-efficient Model Adaptation for Vision Transformers