Understanding Task Transfer in Vision-Language Models