Multi-agent Inverse Reinforcement Learning (MAIRL) aims to recover agent reward functions from expert demonstrations. We characterize the feasible reward set in Markov games, identifying all reward functions that rationalize a given equilibrium. However, equilibrium-based observations are often ambiguous: a single Nash equilibrium can correspond to many reward structures, potentially changing the game's nature in multi-agent systems. We address this by introducing entropyregularized Markov games, which yield a unique equilibrium while preserving strategic incentives. For this setting, we provide a sample complexity analysis detailing how errors affect learned policy performance. Our work establishes theoretical foundations and practical insights for MAIRL.

Inverse Reinforcement Learning (IRL) deals with the problem of deducing a reward function that explains the behavior of an expert agent who is assumed to act in an underlying unknown task. Recent works have studied the IRL problem from the perspective of recovering the, i.e., the class of reward functions that are compatible with a unique optimal expert. However, in several problems of interest it is possible to observe the behavior of multiple experts with different degree of optimality (e.g., racing drivers whose skills ranges from amateurs to professionals). For this reason, in this work, we focus on the reconstruction of the feasible reward set when, in addition to demonstrations from the optimal expert, we observe the behavior of multiple . Given this problem, we first study the theoretical properties showing that the presence of multiple sub-optimal experts, in addition to the optimal one, can significantly shrink the set of compatible rewards, ultimately mitigating the inherent ambiguity of IRL.Furthermore, we study the statistical complexity of estimating the feasible reward set with a generative model and analyze a uniform sampling algorithm that turns out to be minimax optimal whenever the sub-optimal experts' performance level is sufficiently close to that of the optimal expert.

In several problems of interest, it is possible to observe the behavior of multiple agents with different degrees of expertise .

Estimating the unknown reward functions driving agents' behaviors is of central interest in inverse reinforcement learning and game theory. To tackle this problem, we develop a unified framework for reward function recovery in two-player zero-sum matrix games and Markov games with entropy regularization, where we aim to reconstruct the underlying reward functions given observed players' strategies and actions. This task is challenging due to the inherent ambiguity of inverse problems, the non-uniqueness of feasible rewards, and limited observational data coverage. To address these challenges, we establish the reward function's identifiability using the quantal response equilibrium (QRE) under linear assumptions. Building upon this theoretical foundation, we propose a novel algorithm to learn reward functions from observed actions. Our algorithm works in both static and dynamic settings and is adaptable to incorporate different methods, such as Maximum Likelihood Estimation (MLE). We provide strong theoretical guarantees for the reliability and sample efficiency of our algorithm. Further, we conduct extensive numerical studies to demonstrate the practical effectiveness of the proposed framework, offering new insights into decision-making in competitive environments.

The field of inverse constrained RL, which seeks to infer constraints from expert data, is a promising step in this direction.

Inverse Reinforcement Learning (IRL) deals with the problem of deducing a reward function that explains the behavior of an expert agent who is assumed to act optimally in an underlying unknown task. Recent works have studied the IRL problem from the perspective of recovering the feasible reward set, i.e., the class of reward functions that are compatible with a unique optimal expert. However, in several problems of interest it is possible to observe the behavior of multiple experts with different degree of optimality (e.g., racing drivers whose skills ranges from amateurs to professionals). For this reason, in this work, we focus on the reconstruction of the feasible reward set when, in addition to demonstrations from the optimal expert, we observe the behavior of multiple sub-optimal experts. Given this problem, we first study the theoretical properties showing that the presence of multiple sub-optimal experts, in addition to the optimal one, can significantly shrink the set of compatible rewards, ultimately mitigating the inherent ambiguity of IRL.Furthermore, we study the statistical complexity of estimating the feasible reward set with a generative model and analyze a uniform sampling algorithm that turns out to be minimax optimal whenever the sub-optimal experts' performance level is sufficiently close to that of the optimal expert.

Multi-agent Reinforcement Learning has gathered significant interest in recent years due to its ability to model scenarios involving interacting agents. Notable successes have been achieved in domains such as autonomous driving (Shalev-Shwartz et al., 2016; Zhou et al., 2020), internet marketing (Jin et al., 2018), multi-robot control (Dawood et al., 2023), traffic control (Wang et al., 2019), and multi-player games (Baker et al., 2019; Samvelyan et al., 2019). All these applications require carefully designed reward functions, which is challenging even in single-agent settings (Amodei et al., 2016; Hadfield-Menell et al., 2017) and becomes more complex in multi-agent environments where each agent's reward function must be tailored to their specific, potentially different, goals. In many scenarios, it is possible to observe an expert demonstrating optimal behavior, yet the underlying reward function guiding this behavior remains unknown. This is where IRL (Ng and Russell, 2000) becomes crucial. IRL aims to recover feasible reward functions that can rationalize the observed behavior as optimal. However, the initial work in IRL revealed a fundamental challenge: the problem is ill-posed because multiple reward functions can potentially explain the same behavior. To address this, subsequent research has focused on reformulating the IRL problem to make it more practical and applicable in real-world settings (Abbeel and Ng, 2004; Ziebart et al., 2008; Ramachandran and Amir, 2007; Ratliff et al., 2006; Levine et al., 2011). Translating IRL to the multi-agent setting introduces new challenges, particularly regarding the concept of optimality, as each agent's strategy depends on the strategies of all other agents.