Efficient Fine-Tuning with Domain Adaptation for Privacy-Preserving Vision Transformer