Cooperative multi-agent reinforcement learning (MARL) has been proposed for multi-agent collaborations toaccomplish manychallenging tasks[1,2,3,4].

Table 1: Payoff matrix of the one-step multi-state non-monotonic cooperative matrix game and reconstructed resultsfromcorresponding baselines.

To enable decentralized execution, we alsoderivefactorized per-agentpolicies inspired byamaximum-entropyMARL framework.

In this paradigm of centralised training for decentralised execution, QMIX [25] is a popular Qlearning algorithm with state-of-the-art performance ontheStarCraft Multi-Agent Challenge [26]. QMIX represents the optimal joint action value function using a monotonicmixing function of per-agent utilities.