ALP: Action-Aware Embodied Learning for Perception

Although models trained on these datasets have shown strong performance in a wide variety of tasks such as classification, detection, and segmentation, they fundamentally are unable to generalize to an ever-evolving world due to constant out-of-distribution shifts of input data. Therefore, instead of training on fixed datasets, can we approach learning in a more human-centric and adaptive manner? In this paper, we introduce Action-Aware Embodied Learning for Perception (ALP), an embodied learning framework that incorporates action information into representation learning through a combination of optimizing a reinforcement learning policy and an inverse dynamics prediction objective. Our method actively explores in complex 3D environments to both learn generalizable task-agnostic visual representations as well as collect downstream training data. We show that ALP outperforms existing baselines in several downstream perception tasks. In addition, we show that by training on actively collected data more relevant to the environment and task, our method generalizes more robustly to downstream tasks compared to models pre-trained on fixed datasets such as ImageNet. A vast majority of current vision models are fueled by learning from a passively curated dataset. They achieve good performance on new inputs that are close to the training distribution but fail to generalize across changing conditions. This issue is mitigated partially by performing pre-training on a large dataset like ImageNet or COCO. However, these are still static snapshots of objects which do not allow for richer learning, and pre-trained models still fail to enable generalization to new environments and tasks that are semantically unrelated to those in the large-scale datasets. A natural approach to address these issues is to leverage active agents in an embodied environment. Intuitively, learning representations from exploration in the environment allows us to learn from large scale data that is relevant to the task at hand; active exploration can find maximally informative or diverse instances. A similar argument can be made for active data collection of labeled data for downstream fine-tuning.