Probing Visual Language Priors in VLMs