Continuous Control with Action Quantization from Demonstrations

With several successes on highly challenging tasks including strategy games such as Go [Silver et al., 2016], StarCraft [Vinyals et al., 2019] or Dota 2 [Berner et al., 2019] as well as robotic manipulation [Andrychowicz et al., 2020], Reinforcement Learning (RL) holds a tremendous potential for solving sequential decision making problems. RL relies on Markov Decision Processes (MDP) [Puterman, 2014] as its cornerstone, a general framework under which vastly different problems can be casted. There is a clear separation in the class of MDPs between the finite discrete action setup, where an agent faces a finite number of possible actions, and the continuous action setup, where an agent faces an infinite number of actions. The former is arguably simpler, since exploration is more manageable with a finite number of actions, and computing the maximum of the action-value function is straightforward (and implicitly defines a greedily-improved policy). In the continuous action setup, the parametrized policy either directly optimizes the expected value function that is estimated through Monte Carlo rollouts [Williams, 1992], which makes it demanding in interactions with the environment, or tracks the maximum of the bootstrapped value function [Konda and Tsitsiklis, 2000] hence introducing additional sources of approximations. Therefore, a natural workaround consists in turning a continuous control problem into a discrete one. The simplest approach is to naively (e.g.