On Multi-Agent Inverse Reinforcement Learning

Multi-agent Reinforcement Learning has gathered significant interest in recent years due to its ability to model scenarios involving interacting agents. Notable successes have been achieved in domains such as autonomous driving (Shalev-Shwartz et al., 2016; Zhou et al., 2020), internet marketing (Jin et al., 2018), multi-robot control (Dawood et al., 2023), traffic control (Wang et al., 2019), and multi-player games (Baker et al., 2019; Samvelyan et al., 2019). All these applications require carefully designed reward functions, which is challenging even in single-agent settings (Amodei et al., 2016; Hadfield-Menell et al., 2017) and becomes more complex in multi-agent environments where each agent's reward function must be tailored to their specific, potentially different, goals. In many scenarios, it is possible to observe an expert demonstrating optimal behavior, yet the underlying reward function guiding this behavior remains unknown. This is where IRL (Ng and Russell, 2000) becomes crucial. IRL aims to recover feasible reward functions that can rationalize the observed behavior as optimal. However, the initial work in IRL revealed a fundamental challenge: the problem is ill-posed because multiple reward functions can potentially explain the same behavior. To address this, subsequent research has focused on reformulating the IRL problem to make it more practical and applicable in real-world settings (Abbeel and Ng, 2004; Ziebart et al., 2008; Ramachandran and Amir, 2007; Ratliff et al., 2006; Levine et al., 2011). Translating IRL to the multi-agent setting introduces new challenges, particularly regarding the concept of optimality, as each agent's strategy depends on the strategies of all other agents.