Refining Skewed Perceptions in Vision-Language Models through Visual Representations