Appendix for Multiagent Q-learning with Sub-Team Coordination