AdvLoRA: Adversarial Low-Rank Adaptation of Vision-Language Models