Task Bias in Vision-Language Models