Diffusion Augmented Agents: A Framework for Efficient Exploration and Transfer Learning

We introduce Diffusion Augmented Agents (DAAG), a novel framework that leverages large language models, vision language models, and diffusion models to improve sample efficiency and transfer learning in reinforcement learning for embodied agents. DAAG hindsight relabels the agent's past experience by using diffusion models to transform videos in a temporally and geometrically consistent way to align with target instructions with a technique we call Hindsight Experience Augmentation. The framework reduces the amount of rewardlabeled data needed to 1) finetune a vision language model that acts as a reward detector, and 2) train RL agents on new tasks. We demonstrate the sample efficiency gains of DAAG in simulated robotics environments involving manipulation and navigation. Our results show that DAAG improves learning of reward detectors, transferring past experience, and acquiring new tasks - key abilities for developing efficient lifelong learning agents. The most recent notable breakthroughs in AI have come from the combination of large models trained on enormous datasets (Firoozi et al., 2023; Brown et al., 2020; Hoffmann et al., 2022; Reed et al., 2022; Gemini-Team, 2023). However, despite efforts to scale up data collection (Collaboration, 2023; Reed et al., 2022; Bousmalis et al., 2023), data in embodied AI settings is still prohibitively scarce because such agents need to interact with physical environments where sensors and actuators present major bottlenecks (Cabi et al., 2020; Lee et al., 2022). This data scarcity issue is especially pronounced in reinforcement learning scenarios, where rewards are often sparse or completely absent in realistic settings (Ecoffet et al., 2021).