Understanding the impact of misspecification in inverse reinforcement learning

In our recent AAAI 2023 paper, Misspecification in Inverse Reinforcement Learning (Skalse and Abate, 2023), we study the question of how robust the inverse reinforcement learning problem is to misspecification of the underlying behavioural model (namely, how the agent's preferences relate to its behaviour). We provide a mathematical framework for reasoning about this question, and use that framework (based on equivalence classes and orders) to derive necessary and sufficient conditions describing what types of misspecification each of the standard behavioural models are (or are not) robust to. Moreover, we provide several results and formal tools, which can be used to study the misspecification robustness of any behavioural models that may be newly developed. Below, we will first explain the motivation for this work. Then, we will explain our results and, finally, describe ways to extend them. Inverse reinforcement learning (IRL) is an area of machine learning concerned with inferring what objective an agent is pursuing, based on the actions taken by that agent.