What's Left? Concept Grounding with Logic-Enhanced Foundation Models

Recent works such as VisProg and ViperGPT have smartly composed foundation models for visual reasoning--using large language models (LLMs) to produce programs that can be executed by pre-trained vision-language models. However, they operate in limited domains, such as 2D images, not fully exploiting the generalization of language: abstract concepts like " " can also be grounded in 3D, temporal, and action data, as in moving to your . This limited generalization stems from these inference-only methods' inability to learn or adapt pre-trained models to a new domain.