A Appendix

Algorithm 1 shows the execution rules of parallel programs. Terminate the program if no subsequent subroutine exists. Compute the cost of each possible allocation based on the auxiliary functions. The common hyperparameters are listed below. Name V alue learning rate 3e-4 training steps 10M update batch size 256 number of rollout threads 8 rollout buffer size 4096 8 weight of value loss 0.1 weight of policy loss 1 weight of entropy loss 0.01 In cooperative settings, the goal input of the assistive agent is the leading agent's goal.