Physically Grounded Vision-Language Models for Robotic Manipulation