ROS-X-Habitat: Bridging the ROS Ecosystem with Embodied AI

Since the earliest days of robotics, researchers have sought to build embodied agents to perform a variety of jobs, such as assistive tasks in factories [Oliff et al., 2020] or wildfire surveillance [Julian and Kochenderfer, 2019]. Following tremendous advancements in deep learning and convolutional neural networks in the past decade, researchers have been able to develop reinforcement learning (RL)-based embodied agents that interact with the real world on the basis of sensory observations. Software platforms such as OpenAI Gym [Brockman et al., 2016], Unity ML-Agents Toolkit [Juliani et al., 2018], and AI Habitat [Savva et al., 2019] have emerged to address the community's need for training and evaluating RL-based embodied agents end-to-end. Our research group was particularly intrigued by the AI Habitat platform, which offers a high-performance, photorealistic simulator, access to a sizeable library of visually-rich scanned 3D environments, and a modular software design. However, even though these platforms allow roboticists to reuse existing RL algorithms and train agents in simulators with ease, there is a critical step to using them for embodied agents which is only partially addressed: Connecting the trained agent with a real robot. Ideally, after training an RL agent in simulation one would like to take advantage of the extensive set of tools and knowledge from the robotics community to make it easy to embody that agent. One particularly popular tool from the robotics community is ROS, a robotics-focused middleware platform with extensive support for classical robotic mapping, planning and control algorithms ([mov, dwa]) as well as drivers for a wide variety of compute, sensing and actuation hardware. But ROS support for directly training an RL agent is limited, and Gazebo-- the standard simulation environment used for ROS systems-- cannot match the level of photorealism or simulation speed of tools specifically designed to train large-scale RL agents [Liang et al., 2019].