ManipBench: Benchmarking Vision-Language Models for Low-Level Robot Manipulation

One long-standing goal in robotics is to train a "generalist" robot capable of performing diverse tasks, particularly robot manipulation. A promising paradigm for this is to leverage the broad knowledge in Vision-Language Models (VLMs) such as GPT -4 [1] and Gemini [2]. While the community has used VLMs to achieve great generalization in domains like computer vision and natural language processing, robotics faces unique challenges with requiring either difficult-to-scale physical real-world interaction data or simulation data with sim-to-real gaps, making it challenging for VLMs to act as low-level planners. However, recent work has extensively explored incorporating these "foundation" models [3] such that they can generate low-level trajectories executable by an embodiment [4, 5, 6, 7]. This direction is especially important because it offers a path to bypass large-scale, task-specific data collection by leveraging general-purpose pre-trained models. Beyond improving scalability, this enables faster deployment in open-world settings where generalization to unseen tasks and objects is critical. It remains unclear, however, which is the optimal foundation model for a "VLM agent" in tasks like fabric or articulated object manipulation, and how VLMs perform in low-level reasoning tasks required for manipulation. Motivated from these questions, we propose ManipBench: a novel open-source benchmark to evaluate how well VLMs understand the low-level effect of a robot's action on its environment (see Figure 1). While there exist benchmarks to evaluate VLMs for robotics [8, 9, 10, 11, 12, 13, 14, 15, 16], our approach and benchmark differ significantly along axes such as task diversity, model diversity, and particularly our novel multiple-choice question (MCQ) based evaluation design, which efficiently assesses the low-level reasoning capabilities of VLMs without requiring trajectory rollouts, as detailed in Table 1.