Robot Navigation Using Physically Grounded Vision-Language Models in Outdoor Environments