A Mathematical Details

In Section 3.1, the difference between the performance of two joint policies is expressed as follows: In Section 3.1, we claim that We represent the policy using its parameter, i.e. From Proposition 4.7 in (Levin and Peres, 2017), if we have two distributions Then, the following can be derived using Eq. Now we provide a detailed proof. Section 3.2 mentions that there exists a risk of high variance in estimating the policy gradient when Now we use mathematical induction to prove the fact. In Section 3.3, the difference between CoPPO and MAPPO is simplified to the difference between Similar to Appendix A.5, the decentralized policies can be viewed independently, thus The details of our CoPPO algorithm are given in Algorithm 1.