Control-Tutored Reinforcement Learning: an application to the Herding Problem

EXTENDED ABSTRACT Model-free reinforcement learning (or simply reinforcement learning, RL, in what follows) is increasingly used in applications to solve a wide variety of control problems (Kober et al., 2013; Garcıa and Fern andez, 2015; Cheng et al., 2019). The lack of requiring a formal model of the plant renders it appealing for a heuristic, low-cost control design approach that can be easily implemented and adapted to different situations. As a tradeoff, learning processes often require a long training phase where the controller agent learns by trial-and-error how the plant responds to different control actions, and what actions to take to steer its behavior in a desired manner. This problem is particularly relevant when using tabular methods, such as Q-learning, in those situations where reinforcement learning is applied to control dynamical systems defined in continuous spaces (Lillicrap et al., 2019). It is therefore desirable to enhance the learning process by encoding some qualitative knowledge of the system dynamics via appropriate models.