A Bellman Equations for Markov Games

In this section, we present the Bellman equations for different types of values in Markov games. Recall the definition for CCE in our main paper (4), we restate it here after rescaling. First, a CCE always exists since a Nash equilibrium for a generalsum game with payoff matrices (P, Q) is also a CCE defined by (P, Q), and a Nash equilibrium always exists. Third, a CCE in general-sum games needs not to be a Nash equilibrium. However, a CCE in zero-sum games is guaranteed to be a Nash equalibrium.