Weaknesses: My main concern is that results seem to be contradictory to what the authors claimed as the benefit of leveraging GPU accelerations. Specifically, in the "impact statement" the authors described CuLE can "provide access to an accelerated training environment to researchers with limited computational capabilities," but the results show the acceleration won't take into effect unless you use more computation---Figure 2, CuLE runs slower than OpenAI when using a fewer number of environments. If someone can only afford to run 100 environments, would this mean CuLE is not useful here? The limitation of the memory has been noted in the paper which is good. I was confused when looking at Table 3. First, why is there no 120 envs experiment for CuLE?

There was a consensus by the reviewers that this paper should be accepted. The paper provides a Cuda implementation of the ATARI simulator which allows for the running of RL experiments on GPUs. This is a solid contribution and has the clear potential to literally accelerate reinforcement learning research.

Additional Feedback: In the definition of the continuity equation, what does "div" stand for? And how it is defined? The definition of Q-hat in (3.1) implies that the activation function sigma is only applied in the first layer of the network. How much harder would the problem be to analyze if the second layer also applied an activation function? I guess dimensions D and d should be closely related, e.g.

The paper presents some new results regarding the convergence of TD and Q-learning when the action-value function is represented by overparameterized neural networks. The theoretical contribution made by this paper is seen as solid. The weakness described by the reviewers are not major and can be addressed in a minor revision and I therefore recommend accepting this paper.

Weaknesses: - While the theoretical results seem correct, it is not clear to me the advantages of this approach compared to previous work, in particular, gradient Q-learning (GQ). On line 110, it is written that the assumptions are not as stringent but I am not convinced that this is the case. Could the authors clarify this point? If I am interpreting it correctly, it assumes that we have a fixed replay buffer of data on which we are doing updates, as in the offline batch RL setting. It is not specified which policy is used to collect this data and I would expect certain assumptions on this behavior policy.

This paper presents a new objective and an algorithm, which is similar to DQN, that optimises for that objective. Similar to prior work (GTD, TDC, GQ), the algorithm is shown to be convergent under linear function approximation. Because the objective is different, the paper could have better illustrated what this means in terms of the quality of the fixed point the new algorithm converges to - this is only discussed in detail in a special case of a diagonal feature covariance matrix. The author response did not lift this concern, and it remains unclear whether the new algorithm has major benefits over existing related work. The experiments were deemed somewhat insufficient to fully convince the reviewers of this.

The paper addresses the problem of learning a model of Atari 2600 games (a popular testbed for reinforcement learning algorithms), in other words predicting future frames conditioned on action input. This is a challenging problem and its solution is a useful tool to build better controllers. The paper is clear and well-structured, and has convincing experiments (and videos). The model is a CNN (with a fully-connected layer) followed by multiplicative interactions with an action vector, followed by convolution decoding layers. The recurrent version has an LSTM layer added after the CNN.

Weaknesses: The empirically evaluation misses relevant baselines, making it quite hard to evaluate the usefulness of ESRL in comparison to prior approaches. The main algorithm (Algo 1) incorporates the use of majority voting and hypothesis testing in addition to learning multiple Q-estimates based on K sampled MDPs. Furthermore, based on the figure captions, K seems to be large (250 for Riverswim, 500 for Sepsis) and it seems unfair to use a single DQN model. A *naive* baseline would be to use the ensemble of these K Q-estimates and simply use their mean for selecting actions: this *quantifies* the empirical benefit from hypothesis testing. This should be discussed in the paper as well as empirically compared to as should be made as this is a simple way to incorporate value uncertainty in offline RL. 3. As mentioned in the paper, ESRL can deviate from the behavior policy when required or stick to it depending on the hypothesis testing.

This paper proposes an interesting way to use hypothesis testing as a solution to use expert knowledge for offline RL. The proposed approach is exciting and good enough to be published at NeurIPS. The experimental results are interesting, as well. However, the authors should address the concerns on the presentation and theoretical results raised by Reviewer 1 in the camera-ready version of the paper. At the very least, discussing it is the limitation of the approach in the paper's conclusion.

Weaknesses: More attention should be paid for teasing out differences between V and R learning, with intermittent initial rewards being essentially the only example. Although it is impressive that new VTA recording data is presented in the paper, I don't feel that the result is particularly helpful - it only shows that VTA activity doesn't contradict R-learning model, but it does not really provide specific support for it. It should be possible to design different tasks/protocols under which the two formalisations would have substantially different TD errors, which could help tease out biological correlates of the two models. Furthermore, it would be nice to see more details of parameter estimation and the resulting best-fitting parameter values, which if done properly, may allow to achieve not only a qualitative but also a better quantitative fit between Figure 1E and Figure 1D (as well as between Figure 1D and Figure 1B). As the models have multiple parameters substantially affecting performance, the two models should be compared under best-fitting parameters and should include formal measures like AIC, not just qualitative fits. Of course model universality regardless of parameters is helpful, but quantitative fit is equally important.