UAD: Unsupervised Affordance Distillation for Generalization in Robotic Manipulation

Figure 1: Unsupervised Affordance Distillation (UAD) extracts affordance annotations from large pre-trained models and distills them into a task-conditioned affordance model, which is capable of predicting fine-grained affordance in open-world scenes with open-ended instructions, enabling diverse generalization properties in downstream policy learning. Abstract -- Understanding fine-grained object affordances is imperative for robots to manipulate objects in unstructured environments given open-ended task instructions. However, existing methods of visual affordance predictions often rely on manually annotated data or conditions only on a predefined set of tasks. We introduce Unsupervised Affordance Distillation (UAD), a method for distilling affordance knowledge from foundation models into a task-conditioned affordance model without any manual annotations . Training only a lightweight task-conditioned decoder atop frozen features, UAD exhibits notable generalization to in-the-wild robotic scenes and to various human activities, despite only being trained on rendered objects in simulation. Using affordance provided by UAD as the observation space, we show an imitation learning policy that demonstrates promising generalization to unseen object instances, object categories, and even variations in task instructions after training on as few as 10 demonstrations. Understanding the affordances of objects underpins a robot's capability to perform purposeful interactions in unstructured environments [1-3]. Given an open-ended task instruction specified in natural language, a robot must first identify the action possibilities afforded by the environment based on its visual perception. In particular, this understanding should extend beyond objects or object parts to encompass fine-grained details down to the level of pixels. While learning visual affordances from manually annotated datasets with closed vocabulary has been extensively explored in the literature [4-9], scaling affordance learning to open-world scenarios conditioned on free-form task instructions remains a long-standing challenge. Vision-language models (VLMs) have demonstrated the ability to internalize world knowledge by pretraining on large-scale image-text datasets [10, 11]. Recent works also suggest that they encode affordance knowledge in the language space [12] (e.g., "handle should be grasped for opening drawers"). However, the effective grounding of this knowledge in the continuous spatial domain remains an open question. In contrast, self-supervised vision models [13, 14] provide general-purpose pixel-level features that capture low-level structures of objects.