Pixels Versus Priors: Controlling Knowledge Priors in Vision-Language Models through Visual Counterfacts