Maximum Causal Entropy Correlated Equilibria for Markov Games

In this work, we present maximum causal entropy correlated equilibria, a new solution concept that we apply to Markov games. This contribution extends the existing solution concept of maximum entropy correlated equilibria for normal-form games to settings with elements of dynamic interaction with a stochastic environment by employing the recently developed principle of maximum causal entropy. This solution concept is justified for two purposes: as a mechanism for prescribing actions, it reveals the least additional information about the agents' motives possible; and as a predictive estimator of actions for a group of agents assumed to behave according to an unknown correlated equilibrium, it has the fewest additional assumptions and minimizes worst-case action prediction log-loss. Importantly, equilibria for this solution concept are guaranteed to be unique and Markovian, enabling efficient algorithms for finding them.