Summary and Contributions: [UPDATE] I thank the authors for their response. I agree that the empirical results (for the settings considered in the main paper and the additional results provided in the rebuttal) are convincing/impressive. But in the theoretical/algorithmic front, I'm still not convinced. Especially: [1] lines 4-16 in the rebuttal: I still think that Theorem 1 imposes strong restriction of the class of MDPs (even if it relaxes the restriction on the expert policy distribution): not necessarily all the MDPs should satisfy such condition over long-term Q-value. Consider an example: action space that contains only two actions A {a,b}, state s, and a greed/deterministic expert policy s.t.

Reviewers generally agreed that this paper proposes a novel IRL method that leverages the assumption that the expert demonstration is following a boltzmann distribution.

The paper presents a new technique for learning feature matching using reinforcement learning. Strengths are good experimental results showing the technique is effective in several problems. Weaknesses are some concerns about positioning (motivation, related work), but these are addressed in the rebuttal. The final reviews are uniformly positive. This is a clear accept.

Summary and Contributions: Conventionally when rollout-based MBRL algorithms apply an optimistic exploration strategy like UCB, aleatoric and epistemic uncertainty are often conflated into a single pointwise measure of uncertainty at each state in the rollout sequence. This submission proposes a novel augmented policy class that explicitly interacts with the model's epistemic uncertainty to hypothesize the best possible outcome for any particular action sequence. In addition to proof-of-concept experiments on easy Mujoco control tasks, the authors provide regret bounds for their exploration strategy applied to purely rollout-based MBRL methods, including a sublinear regret bound for GP dynamics models. My greatest concern with this submission lies with the reproducibility of the results. There is no mention of code, and simple, crucial implementation details are missing.

In particular they convert epistemic uncertainty into "hallucinated controls" that are optimized, thereby leading to optimistic behavior.

Additional Feedback: COMMENTS AFTER REBUTTAL Thank you for your response. However, in this paper's case I find that the significance of the paper (i.e., support for your claim that "theoretical results provided in this work are important on their own") is severely lacking without experiments showing a link between this theory and an algorithm's performance in terms of measures like running time, number of 1-step Bellman backups, etc. ***Note: this is not a claim that every theoretical paper needs experiments; it applies only to this specific work, due to the theory issues mentioned in the original review.*** The rebuttal's attempted arguments against providing experiments really miss the mark: -- The rebuttal gives the "Beyond the one step greedy approach in RL" as an example of a paper similar in the degree of its theoretical focus to this submission, but that paper actually has experiments! Light experiments could do the job. That "Beyond the one step greedy approach in RL" paper that you mentioned yourself is a case in point.

Weaknesses: The performance improvement is incremental and needs to be further evaluated. For example, each experiment should be conducted over 5 random seeds, instead of 3 seeds, for a more accurate comparison. Besides, in only 3 out of 8 environments, shown in Figure 2, the proposed method shows clear improvement. And more baseline methods should be considered, such as SAC. So, how does the generalise SIL compare to SIL in the Montezuma's Revenge task?

The author response provided satisfactory answers to the concerns of the reviewers with respect to contraction/bias tradeoff, disconnect between the experimental results and theory, and variance of the estimator. This lead one reviewer to increase their score for this paper, which already had reasonably solid scores.

Summary and Contributions: This paper studies the exploration problem in episodic reinforcement learning with kernel and neural network function approximations. The proposed algorithm is an optimistic version of least-squares value iteration, where the solution to the standard LSVI is further added by a bonus function for exploration. Under assumptions on the underlying RKHS or NTK function classes, the proposed algorithms are shown to achieve a H 2 \sqrt{T} \delta_F regret, where \delta_F depends on the effective dimension of the RKHS or NTK. First, state clearly in the introduction (maybe also abstract) that this paper makes the assumption that the transition model is characterized by the RKHS class -- I think you already did but it doesn't hurt to emphasize it. Also, revise the sentence "propose the first provable efficient RL algorithm [...] without any additional assumptions on the sampling model" (lines 44-46), e.g., by changing the term "sampling model" to be "generative model" or "simulator", as such a term is ambiguous.

This paper studies the exploration problem in episodic reinforcement learning with kernel and neural network function approximations. The authors propose a novel algorithm which is an optimistic version of least-squares value iteration, where the solution to the standard LSVI is further added by a bonus function for exploration. They derive regret bounds for this algorithm for two different function classes: RKHS and NTK. Overall, the technical contribution in this paper seems solid. Some reviewers had some concerns about the assumptions made for the analysis, especially regarding the one assuming that the Bellman optimality update lies in the RKHS.