Can Vision Language Models Infer Human Gaze Direction? A Controlled Study