Using VLM Reasoning to Constrain Task and Motion Planning