Dual Memory Networks: A Versatile Adaptation Approach for Vision-Language Models