A Extended Related Work

We extend our related work section on the following related topics as suggested by reviewers: A.1 Discussion on Ensembles and Distributional RL In our main text, the estimated values for extremely o.o.d. On the one hand, it's clear that such an assumption holds for the tabular settings, that un-visited On the other hand, we acknowledge it as a mild assumption that there always exists o.o.d. The key insight we want to emphasize in Section 4.3.1 is that for frequently visited state-action pairs, Network Structure Our implementation of TD3, BCQ and CQL are based on code released by the authors, without changing hyper-parameters. Our code is provided in the supplementary materials, and will be made public available. Our implementation of BCQ and CQL are both based on the code provided by the authors.