The reviewers appreciated the rebuttal that provided some additional insights.

Weaknesses: A few comments that are needed to be addressed: 1) The first comment is about the presentation of the derivations. There are steps in the appendix, and also in the main text that are skipped. Some of them took me a while to rederive, some I couldn't spend more time to rederive. Some steps are also taken as granted in the main text. It is useful to elaborate on them more.

Summary and Contributions: ---post author response--- Thank you for the response! The clarifications to the table have improved my understanding of the results. While I think that the results are strong, the discussion section is jumbled/unclear, and intuition of some of the design decisions are lacking and give an'ad hoc' impression. Clarifications for this are adequately mentioned in the response, and I will increase my score to a 6 assuming the authors will add these clarifications to the final text, as well as make the experimental results section more more clear. This work proposes a batch deep RL algorithm called BAIL. It essentially trains a policy using imitation learning with samples collected from state-action pairs whose (Monte Carlo) returns are from what the authors define as the upper envelope of the data.

The authors agreed that the paper makes good contributions to batch RL, and the rebuttal has been very helpful. Some concerns around the empirical evaluation remain, but the paper makes a good contribution. Please make sure that the revised version of the paper actually reflects the rebuttal and reviewer recommendations.

Weaknesses: I have some concerns and questions: - In order to come up with an efficiently-implementable algorithm, for DORL the authors construct an optimistic MDP following a very simple construction. This construction only considers error bounds and completely ignores the value function. So, while the proof claims the optimism is guaranteed, I believe that the resulting optimistic MDP is overly-optimsitic, and to favor computational efficiency, this way one may sacrifice learning efficiency to a large extent. Indeed, the idea of optimizing over a finite set of MDPs (in lieu of the bounded-parameter MDP) is nice. However, I believe the current construction that completely ignores the value function is too naive to work in practice.

After discussing with the reviewers, we have decided to propose acceptance for the paper. Nonetheless, I would like the stress that the current submission has a number of critical aspects that need to be addressed by the authors to make the paper more solid. I strongly encourage the authors to read reviewers' comments and focus on improving along the following directions: - Clarity: Reviewers all agree that the paper could be improved in writing to make it more accessible to an audience that is not strictly familiar with the factored MDP formalism and/or the technicalities behind UCRL proofs. In particular, it is unclear how the parameter c has been chosen and why it takes significantly different values for different algorithms. It would be helpful to see the performance as c changes. - The way optimism is obtained is probably not very tight and it may cause over exploration for a long time.

Weaknesses: The paper may need to be improved to address a few important issues, as detailed below. Why is it important to enhance population-wide behavioral diversity? Intuitively I can understand the potential benefits related to deep exploration and learning stability. However, theoretically I cannot link the benefits straightforwardly to the proposed use of kernel function and the kernel matrix determinant. Theorem 3.3 states that when lambda is set properly, the population will contain M distinct optimal policies.

This paper focuses on an interesting problem of maintaining diversity in a set of agents. The paper formalizes the problem clearly, and the initial results presented are positive and support the paper's claims. The paper is fairly well written. Among the aspects of the paper that could be improved upon, the experimental section lacks many details and no ablation studies are performed, it is not clear how the new technique compares from a computational cost perspective, and some of the implications of the assumptions made (e.g., on the scope of problems to which this approach can reasonably be applied) are not clearly stated. Overall, the paper was found to make a sufficient contribution and the final recommendation is to accept.

I would just like to confirm my understanding of the algorithmic contributions of this work. As far as I understand, Jin et al [2019] propose a learning algorithm for the standard RL case with linear function approximation in linear MDPs. Then Jin et al [2020] propose a method for efficient exploration in the reward-free RL case. This is for normal MDPs but in the tabular setting. In that work, exploration is achieved by constructing a reward function where the reward is 1 for states that are "significant", and 0 otherwise, and then solving the resulting task with an efficient learning algorithm.

The authors study sequential decision processes without reward function. The goal is to learn the transition dynamics such that various reward functions could be optimised efficiently in the future. The authors extend recent work to the linear function approximation case. They provide an analysis of the sample complexity, and show that while for linear MDPs complexity is polynomial, this is not true for MDPs with a linear optimal value functions, providing insight on the hardness of this second class of problems. The strengths of the paper are the theoretical development of the algorithm and the lower bound for MDPs with linear optimal Q functions.