Model-based Reinforcement Learning with a Hamiltonian Canonical ODE Network

Model-based reinforcement learning usually suffers from a high sample complexity in training the world model, especially for the environments with complex dynamics. To make the training for general physical environments more efficient, we introduce Hamiltonian canonical ordinary differential equations into the learning process, which inspires a novel model of neural ordinary differential auto-encoder (NODA). NODA can model the physical world by nature and is flexible to impose Hamiltonian mechanics (e.g., the dimension of the physical equations) which can further accelerate training of the environment models. It can consequentially empower an RL agent with the robust extrapolation using a small amount of samples as well as the guarantee on the physical plausibility. Theoretically, we prove that NODA has uniform bounds for multi-step transition errors and value errors under certain conditions. Extensive experiments show that NODA can learn the environment dynamics effectively with a high sample efficiency, making it possible to facilitate reinforcement learning agents at the early stage. Reinforcement learning has obtained substantial progress in both theoretical foundations (Asadi et al., 2018; Jiang, 2018) and empirical applications (Mnih et al., 2013; 2015; Peters & Schaal, 2006; Johannink et al., 2019). In particular, model-free reinforcement learning (MFRL) can complete complex tasks such as Atari games (Schrittwieser et al., 2020) and robot control (Roveda et al., 2020). However, the MFRL algorithms often need a large amount of interactions with the environment (Langlois et al., 2019) in order to train an agent, which impedes their further applications. Model-based reinforcement learning (MBRL) methods can alleviate this issue by resorting to a model to characterize the environmental dynamics and conduct planning (van Hasselt et al., 2019; Moerland et al., 2020a). In general, MBRL can quench the thirst of massive amounts of real data that may be costly to acquire, by using rollouts from the model (Langlois et al., 2019; Deisenroth & Rasmussen, 2011).