Neural Information Processing Systems http://nips.cc/

We first redefine notation for clarity and then provide the proofs of the results in the main paper. Now we first prove that the iteration in Eq.2 has a fixed point. Proof of Lemma 3.1: Let We present the bound on using empirical Bellman operator compared to the true Bellman operator. The proof can be found in [6]. Proof of Theorem 3.4: Recall that the expression of the V -function iterate is given by: Proof of Theorem 3.6: The proof of this statement is divided into two parts.

Neural Information Processing Systems http://nips.cc/

Model-free deepreinforcement learning (RL)algorithms havebeenwidely used for a range of complex control tasks.

But in the field of reinforcement learning (RL),L2-norm is the most common choice due to its5 efficiency and effectiveness. Thus we adoptL2-norm in the paper to ensure consistency between the objective of6 Andersonacceleration(AA)andthelossofQ-valuefunction(critic).7 Minors.

We observe that part of this difficulty stems from the "signal-to-noise"

Multi-agent systems are characterized by environmental uncertainty, varying policies of agents, and partial observability, which result in significant risks.

Neural Information Processing Systems http://nips.cc/

This section consists of three parts, with each subsequent part building upon the previous one. Appendix A.1 covers the fundamentals of RL, where the actor-critic method is introduced. Appendix A.2 describes the RL algorithm for a single fulfillment agent, which is the proximal policy Appendix A.3 presents the MARL algorithm for the Currently, policy-based methods [Deisenroth et al., 2013] are prevalent because they are compatible with stochastic To sum up, the complete procedure is given in Algorithm 1.Algorithm 1 Heterogeneous Multi-Agent Reinforcement Learning for Order Fulfillment. With regard to the advantage estimator, we set the GAE parameters [Schulman et al., 2016] To highlight how our proposed benchmark differs from existing approaches focused on sub-tasks of order fulfillment, we compare the objectives, observations, and actions in Table 1. It should be noted that multiple formulations exist for each sub-task.