The paper develops a variational inference algorithm for modelling discrete state action MDPs. The model can be used to capture the correlations inherent between the states of an MDP. For doing so, Polya-Gamma auxiliary variables have been used, which has been proposed before. The contribution of the paper is the variational inference algorithm instead of using Block-based Gibbs sampling as in the original model. The paper attacks a very important problem, follows a nice idea and is well executed.

Additional Feedback: *) Is there a reason to mention algorithm 1? it seems algorithm 2 gives improved performance relatively to it. If so, why presenting the two algorithms and not just algorithm 2? *) Although equation 9 can be thought of as a set of n m linear constraints, why the optimization problem is always feasible? Although the authors devoted half a page to explain on this procedure, I feel it is not well explained. Most of the discussion is not devoted to explaining the policy certification procedure. Why for a fixed \mu the best response is not markovian?

After reading the reviews and authors' responses, it seems the only main concern raised is the lack of experiments. My opinion is that while experiments would be nice to have, the lack of experiments is not a significant concern if the theoretical results are strong enough. In my own assessment of the paper, I find the theoretical results to be indeed quite a strong contribution to the field (they provide the first algorithm to match the PAC lower bound, for a problem which has quite a few previous works). The reviewers seem to agree with this point in their reviews. I, therefore, recommend that the paper be accepted.

This paper broadly considers the use of a learned parametric model. Through (1) toy examples, (2) theoretical analysis of a Dyna-like algorithm, and (3) a large scale study of sample-efficient model-free RL, it arrives at the conclusion that "using an imperfect (e.g., parametric) model to generate fictional experiences from truly observed states… should probably not result in better learning." While the individual pieces described above are all valuable, I am not sure this claim is properly qualified. For example: "More generally, if we use a perfect model to generate experiences only from states that were actually observed, the resulting updates would be indistinguishable from doing experience replay. In a sense, replay is a perfect model, albeit only from the states we have observed." I am not sure this is, as stated, exactly true.

There's consensus that this is a well written paper that offers some useful insights about the pros and cons of model-based RL vs. model-free RL with replay buffers. This is an important topic and this paper has the potential to make significant impact. However, the authors are urged to be careful about not making overly general conclusions in the final version of the paper, as this was a concern of one reviewer. Even the title may be too general.

This paper proposes using Bayesian linear regression to get a posterior over successor features as a way of representing uncertainty, from which they sample for exploration. I found the characterization of Randomised Policy Iteration to be strange, as it only seems to apply to UBE but not bootstrapped DQN, With bootstrapped DQN, each model in the ensemble is a value function pertaining to a different policy, thus there is no single reference policy. The ensemble is trying to represent a distribution of optimal value functions, rather than value functions for a single reference policy. Proposition 1: In the case of neural networks, and function approximation in general, it is very unlikely that we will get a factored distribution, so this claim does not seem applicable in general. In fact, in general there should be very high correlation between the q-values between nearby states. Is this claim a direct response to UBE? Also the analysis fixes the policy to consider the distribution of value functions, but this seems to not be how posterior sampling is normally considered, but rather only the way UBE considers it.

From the discussion, the reviewers appreciated the precisions made in the rebuttal. They have indicated what they would like to see improved in a revised version, in particular a clearer presentation.

Strengths: The novelty of the paper is to provide a scalable learning method for average reward settings with guarantee of small performance loss. The work is relevant to a number of real world applications such as social networks, communication networks, transportation networks etc. Following are the highlights of the paper - The problem formulation is clear and despite having so many variables in the proofs, the mathematical notations are wisely chosen and are unambiguous. The proofs appears to be correct and I liked the way few assumptions have been used to provide theoretical guarantees. Overall I think the paper should be accepted for publication.

The paper has been extensively discussed and reviewers agree the paper has merit and the rebuttal brings a lot of clarification on a number of questions identified by the reviewers (e.g. the difference between underlying framework of the proposed method and that of mean field RL). General consensus is to propose acceptance of the paper; reviewers would like the authors to clarify the following in the paper though: In difference to their claim, Theorem 2 does not really depend on Theorem 1, as it only assumes the exponential decay property, which Theorem 1 only widens.

In many real-world scenarios, such as gas leak detection or environmental pollutant tracking, solving the Inverse Source Localization and Characterization problem involves navigating complex, dynamic fields with sparse and noisy observations. Traditional methods face significant challenges, including partial observability, temporal and spatial dynamics, out-of-distribution generalization, and reward sparsity. To address these issues, we propose a hierarchical framework that integrates Bayesian inference and reinforcement learning. The framework leverages an attention-enhanced particle filtering mechanism for efficient and accurate belief updates, and incorporates two complementary execution strategies: Attention Particle Filtering Planning and Attention Particle Filtering Reinforcement Learning. These approaches optimize exploration and adaptation under uncertainty. Theoretical analysis proves the convergence of the attention-enhanced particle filter, while extensive experiments across diverse scenarios validate the framework's superior accuracy, adaptability, and computational efficiency. Our results highlight the framework's potential for broad applications in dynamic field estimation tasks.