A Trust Region Approach for Few-Shot Sim-to-Real Reinforcement Learning

Simulation-to-Reality Reinforcement Learning (Sim-to-Real RL) seeks to use simulations to minimize the need for extensive real-world interactions. Specifically, in the few-shot off-dynamics setting, the goal is to acquire a simulator-based policy despite a dynamics mismatch that can be effectively transferred to the real-world using only a handful of real-world transitions. In this context, conventional RL agents tend to exploit simulation inaccuracies resulting in policies that excel in the simulator but underperform in the real environment. To address this challenge, we introduce a novel approach that incorporates a penalty to constrain the trajectories induced by the simulator-trained policy inspired by recent advances in Imitation Learning and Trust Region based RL algorithms. We evaluate our method across various environments representing diverse Sim-to-Real conditions, where access to the real environment is extremely limited. These experiments include high-dimensional systems relevant to real-world applications. Across most tested scenarios, our proposed method demonstrates performance improvements compared to existing baselines. Reinforcement Learning (RL) is often applied in simulation before deploying the learned policy on real systems (Ju et al., 2022; Muratore et al., 2019; Kaspar et al., 2020; Witman et al., 2019). This approach is considered to be one of the safest and most efficient ways of obtaining a near-optimal policy for complex systems (Jiang et al., 2021; Salvato et al., 2021; Hsu et al., 2023), as many of the challenges of applying RL to real-world systems (Dulac-Arnold et al., 2021) are mitigated. The agent can sample the simulator at will (Kamthe & Deisenroth, 2018; Schwarzer et al., 2021) without having to consider any safety constraints (Garcıa & Fernández, 2015; Achiam et al., 2017) during training. However, simulators of complex systems are often inaccurate. Indeed, many physical laws such as contact forces, material elasticity, and fluid dynamics are difficult to model, leading simulators to rely on approximations (Koenig & Howard, 2004; Todorov et al., 2012).