Appendices

In this subsection, we prove the lemmas stated in the paper. Lemma 3. F or any state s S, we have Var Remark 2, the multi-agent advantage is bounded from both sides. It suffices to prove the first inequality, as the second one is a trivial upper bound. Theorem 2. The COMA and DT estimators of MAPG satisfy Var We rely on this fact in the proofs below. From the decomposition of the estimator's variance, we know that minimisation of the In the paper, we discussed the impracticality of the above baseline.