A Taxonomy for Evaluating Generalist Robot Policies

--Machine learning for robotics promises to unlock generalization to novel tasks and environments. Guided by this promise, many recent works have focused on scaling up robot data collection and developing larger, more expressive policies to achieve this. But how do we measure progress towards this goal of policy generalization in practice? Evaluating and quantifying generalization is the Wild West of modern robotics, with each work proposing and measuring different types of generalization in their own, often difficult to reproduce, settings. In this work, our goal is (1) to outline the forms of generalization we believe are important in robot manipulation in a comprehensive and fine-grained manner, and (2) to provide reproducible guidelines for measuring these notions of generalization. We first propose - Gen, a taxonomy of generalization for robot manipulation structured around visual, semantic, and behavioral generalization. We discuss how our taxonomy encompasses most prior notions of generalization in robotics. Next, we instantiate -Gen with a concrete real-world benchmark based on the widely-used Bridge V2 dataset. We evaluate a variety of state-of-the-art models on this benchmark to demonstrate the utility of our taxonomy in practice. Our taxonomy of generalization can yield many interesting insights into existing models: for example, we observe that current vision-language-action models struggle with various types of semantic generalization, despite the promise of pre-training on internet-scale language datasets. We believe -Gen and our guidelines can improve the dissemination and evaluation of progress towards generalization in robotics, which we hope will guide model design and future data collection efforts. We provide videos and demos at our website stargen-taxonomy.github.io. Learning-based robotics often comes with the promise of generalization. As an example, an ambitious goal is to train a policy on diverse household data so it can enter a new home and fold laundry. This vision has led to many recent works that train robot policies on diverse datasets via imitation learning [1-13] with the hope of broad generalization. For example, if a robot encounters an unseen item of clothing in a new home, it should infer how to fold it using its extensive prior experience. However, in contrast to other domains like language and vision, we have yet to reach a point in robotics where policies can reliably generalize in this manner. In pursuit of reliable and broad generalization, recent work has focused on scaling up data collection [2-4, 14-20] and developing more expressive models [3, 7-13], following the successes of other machine learning fields. Although these advances have led to more capable policies that certainly generalize to some novel scenarios, it is often unclear from existing evaluations how generalist these policies truly are.