CARoL: Context-aware Adaptation for Robot Learning

--Using Reinforcement Learning (RL) to learn new robotic tasks from scratch is often inefficient. Leveraging prior knowledge has the potential to significantly enhance learning efficiency, which, however, raises two critical challenges: how to determine the relevancy of existing knowledge and how to adap-tively integrate them into learning a new task. CARoL incorporates context awareness by analyzing state transitions in system dynamics to identify similarities between the new task and prior knowledge. It then utilizes these identified similarities to prioritize and adapt specific knowledge pieces for the new task. Additionally, CARoL has a broad applicability spanning policy-based, value-based, and actor-critic RL algorithms. The simulations include CarRacing and LunarLander environments, where CARoL demonstrates faster convergence and higher rewards when learning policies for new tasks. In real-world experiments, we show that CARoL enables a ground vehicle to quickly and efficiently adapt policies learned in simulation to smoothly traverse real-world off-road terrain. In recent years, Reinforcement Learning (RL) approaches have achieved remarkable success in advanced robotic control and complex task learning in dynamic environments, enabling applications across various domains, such as autonomous navigation [36, 38], manipulation [28, 42], and human-robot interaction [23]. Despite these advancements, RL methods are typically computationally demanding, as they rely on repeated trial-and-error exploration to discover high-reward outcomes. Knowledge fusion [2] and adaptation [24, 35] provide promising approaches to address the inefficiency of RL. They leverage knowledge (such as a learned control policy, approximated value function, etc.) from previously explored tasks to accelerate training on new tasks, eliminating the need to train from scratch for every scenario. For example, consider a vehicle navigating highly complex off-road terrain as shown in Figure 1. Suppose the vehicle has undergone extensive training in several existing environments, it should ideally be capable of adapting to a new type of terrain by utilizing previously learned knowledge.