Multi-Modal Adapter for Vision-Language Models