How Does Vision-Language Adaptation Impact the Safety of Vision Language Models?