This paper presents a benchmark suite for offline RL, along with baseline results on this suite. There is a clear need for such a benchmark suite and all reviewers agree that this submission is a good first step towards fulfilling this need. Only R3 considers that it is not ready yet for acceptance, due to lacking more realistic benchmarks (i.e. Although I agree that this is a valid concern (and I encourage the authors to follow R3's advice and look up potential simulators that may fill up this gap), I also agree with other reviewers that the current state of the proposed benchmark should already be very useful to the research community. As a result, I recommend to accept this paper.

Originality: Considering that multi-view and multi-modal RL papers tend to offer ad-hoc solutions to the problem, this paper's formalization is a nice contribution. Quality & Clarity: The paper is well presented, the math is mostly clear, although some parts aren't obviously translatable to an implementation. In terms of experiments, it seems that many details are lacking, and as far as I call tell, all figures represent a single run of each setting, which is worrisome. Significance: While the contributed framework does seem like a useful formalism, this paper fails to convince me that it actually is: - The proposed experiment in 4.1 creates artificial views which don't seem representative of multimodal settings, in that they all contain the same *information*. It would have been more convincing to feature an experiment where views are truly independent when conditioned on the current state (e.g.

Reviewers are all excited about the extension of model-free and model-based RL to multi-view/multi-modal settings. The authors offer formal formulation of multi-view RL by extending POMDP. The model-free case is rather straight-forward, and the model-based case leads to variational inference. Reviewers however, expressed concern regarding the setup of toy examples and the quality of the evaluation. Overall, the merit of the work outweigh the weakness.

The definition of two neighboring reward functions is provided in Theorem 5. The authors did not explain the motivation of the guarantee of privacy for reward function clearly. It would be better if the authors could interpret the necessities of the privacy of reward function in some real application situations. What is the reason for adding noise like line 19-20 of the Algorithm 1? The definitions of g _k[B][2] in line 4 and g _a[:][1] in line 15 are not given.

This paper proposes a differentially private Q-learning algorithm for RL with continuous observations. This is a nice application of the functional Gaussian noise mechanism, and the paper provides a rigorous privacy and utility analysis. When preparing the final version the authors should fix the presentation issues raised in the reviews, and make sure the paper is properly positioned wrt previous work (eg. BGP'16 used a stricter notion of neighbouring relation between datasets that includes changes in the states and actions, not only rewards).

Weaknesses: Some decisions in the paper are not well motivated, and despite the extensive set of ablations the importance of some choices remains unclear. There are really two separate methodological improvements proposed in this paper: the implicit distributional value function and the semi-implicit policy. These two components might have been better off proposed separately so that they could be studied in more detail. One paper could propose the implicit parameterization of the distributional value function and compare its results to C51 and QR-DQN, while another used a standard expected-value critic with the semi-implicit policy and evaluated in detail the impact of the policy parameterization compared to Gaussian, mixture of Gaussian, and normalizing flow policies. Further complicating matters, there are a lot of bells and whistles in the final method (twin delayed critics, learned temperature, etc).

There was much discussion around the relationship with Tessler et al., which at first seemed quite close. I reached out to the authors for a clarification, as Reviewer 3 had omitted requesting it in their review. After this clarification, the reviewers decided that the work was indeed sufficiently novel and interesting. For the record, the reviewers unanimously appreciated receiving the author clarification (as, I can imagine, the authors appreciated sending it). However, that clarification itself was quite long (5 paragraphs).

The paper proposes a deep reinforcement learning approach for multi-level combinatorial optimization. Reviewers agree on accepting the paper and recognize its novelty and writing standard. The rebuttal addressed successfully concerns raised by reviewers. The experimental part of the paper did not convince the reviewers, though, due to small problem sizes. The recommendation for the paper is accept.

Weaknesses: My main concern about the paper is whether this proposed algorithm is actually implementable due to the specific expression of the (constant) learning rate. I have two concerns: 1. The learning rate depends on t_{mix} in Theorem 1 and on the universal constants c_1 in both Theorem 1 and Theorem 2. How can we compute/approximate t_{mix} in advance? If we cannot, is it sufficient to employ a lower-bound on t_{mix}? Looking at the proofs c_1 is a function of constant c (Equation 55) that in turn derives from Bernstein's inequality (Equation 81) and subsequently \tilde{c} (Equation 84), but its value is never explicitly computed. I am aware that also in [33] the learning rate schedule (that is not constant) depends on \mu_{min} and t_{mix}, but I think the authors should elaborate more on this and explain how to deal with it in practice, if possible.

The reviewers appreciated the efforts made by the authors in the rebuttal, and updated their reviews accordingly. The paper contributions are now clear and important (an improved sample complexity analysis of asynchronous Q-learning, and a novel variance reduction algorithm and its analysis). We recommend the paper for acceptance and encourage the authors to account for the reviewers' comments when preparing the camera-ready version of the paper.