Effective and Efficient Adversarial Detection for Vision-Language Models via A Single Vector