A Appendix

Notice that the Tabular CRR exp objective looks different from the learning rule defined by Eqn. 4. Following Eqn. 8, we see that whenever µ In addition to being safe, we show that each iteration of CRR improves performance. To compute the performance of each agent, as reported in the Tables 2, 3,5, 6 and 7, we adopt the following procedure. We run each agent with three independent seeds. Agent snapshots are made every 50000 learner steps. As discussed in Sec. 3 using K-step returns can hurt the agent's performance To test this hypothesis, we evaluate CRR's (using the binary This objective is similar to the ones used in [27, 7].