Appendix to Weakly Coupled Deep Q-Networks A Proofs