Difference Vector Equalization for Robust Fine-tuning of Vision-Language Models