Rethinking Misalignment in Vision-Language Model Adaptation from a Causal Perspective