Reinforcement Learning with Dynamic Convex Risk Measures

Reinforcement learning (RL) provides a (model-free) framework for learning-based control. RL problems aim at learning dynamics in the underlying data and finding optimal behaviors while collecting data via an interactive process. It differs from supervised learning that attempts to learn classification functions from labeled data, and unsupervised learning that seeks hidden patterns in unlabeled data. During the training phase, the agent makes a sequence of decisions while interacting with the data-generating process and observing feedback in the form of costs. The agent aims to discover the best possible actions by interacting with the environment and consistently updating their actions based on their experience, while often, also taking random actions to assist in exploring the state space - the classic exploration-exploitation trade-off (Sutton and Barto, 2018). In RL, uncertainty in the data-generating process can have substantial effects on performance. Indeed, the environmental randomness may result in algorithms optimized for "on-average" performance to have large variance across scenarios.