The paper provides nice near-optimal sample complexity results for a setting of feature-based MBRL. The results are nontrivial extensions of previous tabular results. On the other hand, it requires a pretty strong anchor-state assumption, which to some extent limits the significance of the results.

The authors propose a novel approach for solving POMDPs by simultaneously learning and solving reward machines. The method relies on building a finite state machine which properly predicts possible observations and rewards. The authors demonstrate that their method outperforms baselines in three different partially observable gridworlds. Overall, I found the paper clear and well motivated. Learning to solve POMDPs is a very challenging problem and any progress or insight has the potential to have a big impact.

Eq. (3), Eq. (5) and its model details) is consistent with the target task. The reward and constraints are reasonably designed. The experimental setting is remarkable (especially the Online Evaluation by simulator and the four proposed evaluation metrics) and the results are positive. However, this paper still has the following minor issues.

The paper describes a constraint augmented RL technique for text generation. The paper is well written. The algorithm is novel and significant.

Originality: The main idea of the paper - avoiding the long horizon problem by computing IS over state distributions rather than trajectories - was already introduced in (Liu et. However, the approach the authors take to leveraging this idea is original. Additionally, there is not yet enough published work on leveraging this potentially important idea (IS over state distribution), and therefore even being the second paper in this direction is still charting new territory. Quality - To the extent I looked at it the theoretical work is solid. I did not go over every equality in the proofs to check for algebraic errors, but I did go through every step in the proofs found in the appendix.

The paper studies the important problem of off-policy policy evaluation in long-horizon MDPs. The setting focuses on small-state, large-action problems. A novel estimator is proposed, whose finite-sample statistical properties are studied. Empirical results show the method is useful, especially in partially observable problems. Reviewers feel the experiment section can be strengthened (e.g., using more domains).

The introduction needs to mention that approaches like the latter *are* available solutions and frame the contribution of the paper rather as one of providing a "better" solution in whichever way the authors feel this is best described (more-efficient, etc.). MINOR COMMENTS: * It seems that else at the beginning of Algorithm 3, line 9 doesn't belong there.

The paper formulates a budgeted Markov decision process (BMDP) able to deal with large search spaces. All reviewers feel the proposed method is novel, interesting and could be an important step in trying to address some existing problems with "modern" RL approaches.

Watter, Manuel, et al. "Embed to control: A locally linear latent dynamics model for control from raw images."

This is especially the case for deep RL approaches that learn both a state abstraction and a model.