4547dff5fd7604f18c8ee32cf3da41d7-Supplemental.pdf

Wecomputethepriorityof eachtrajectoryas ξ = 0.9 maxiξi+0.1 ξ [21],whereξi istheTDerrorperstep.Fromthetraining perspective we have a training loop that continuously samples trajectories from the replay buffer and updates the model based on TD error. The simulation policies are updated to be the training policyevery10gradient steps. Concretely, each of the games played simultaneously has an agent from a set level. Therefore we refer to this policy asRankBot. Similarly, we may expect a color based equivalent of the Rank Bot but in practice we find it difficult to learn such policy naturally.