Partially Observable Markov Decision Processes (POMDPs) provide a robust framework for decision-making under uncertainty in applications such as autonomous driving and robotic exploration. Their extension, $\rho$POMDPs, introduces belief-dependent rewards, enabling explicit reasoning about uncertainty. Existing online $\rho$POMDP solvers for continuous spaces rely on fixed belief representations, limiting adaptability and refinement - critical for tasks such as information-gathering. We present $\rho$POMCPOW, an anytime solver that dynamically refines belief representations, with formal guarantees of improvement over time. To mitigate the high computational cost of updating belief-dependent rewards, we propose a novel incremental computation approach. We demonstrate its effectiveness for common entropy estimators, reducing computational cost by orders of magnitude. Experimental results show that $\rho$POMCPOW outperforms state-of-the-art solvers in both efficiency and solution quality.

We would like a robot to navigate to a goal location while minimizing state uncertainty. To aid the robot in this endeavor, maps provide a prior belief over the location of objects and regions of interest. To localize itself within the map, a robot identifies mapped landmarks using its sensors. However, as the time between map creation and robot deployment increases, portions of the map can become stale, and landmarks, once believed to be permanent, may disappear. We refer to the propensity of a landmark to disappear as landmark evanescence. Reasoning about landmark evanescence during path planning, and the associated impact on localization accuracy, requires analyzing the presence or absence of each landmark, leading to an exponential number of possible outcomes of a given motion plan. To address this complexity, we develop BRULE, an extension of the Belief Roadmap. During planning, we replace the belief over future robot poses with a Gaussian mixture which is able to capture the effects of landmark evanescence. Furthermore, we show that belief updates can be made efficient, and that maintaining a random subset of mixture components is sufficient to find high quality solutions. We demonstrate performance in simulated and real-world experiments. Software is available at https://bit.ly/BRULE.

The major contributions of this paper are that it proves the global convergence of BP(Theorem 1.3) and VI(Theorem 1.2) on ferromagnetic Ising model with a specific initialization, i.e., to initialize variables to be 1. The proof of Theorem 1.2 is based on the fact that the mean-field free energy function, i.e., \Phi(x) is concave on the set S obtained by the update rule, and then we can use Holder's inequality to expand the \Phi(x*) - Phi(x_t) and get the upper bounds. The proof of Theorem 1.3 is based on the fact that the norm of \Phi(v)'s gradient is less than 1(Lemma 3.2), and the properties of variable \mu sandwiched between v 0 and final v T(Lemma 3.5 and Lemma F.1). Other minor contributions include that it provides examples to empirically show the convergence(appendix G) and it shows how to use ellipsoid method to optimize the beliefs(appendix H). I have to admit that I am not familiar with this area, so can only go through a part of the proof, and I am not able to evaluate the originality and quality of this work.

The reviewers liked the results on convergence of belief propagation algorithms for Ising models under certain settings. As a presentational suggestion, they suggest providing more extensive proof sketches in the main section of the paper.

This submission drew a great deal of discussion -- primarily on the point of the role of learning. All reviewers agreed that the approach had the potential to learn interesting, non-trivial things but did not feel the the current experiments demonstrated these effectively -- despite strong performance on the task. Some examples of questions that were not answered by the main draft but came up in the discussion: [Training Data] The training data provides edges in the dependency graph, subgoals, and predicate value -- image pairs. One question was whether the union of the seen dependency graph constituted the entire true underlying graph. Similarly, do all predicate-object pairs occur?

Post rebuttal update: I appreciate the additional explanation for need of overshooting in empirical methods, and the clarity of response regarding stochastic models. The issue I took was with Sec 2.2, that next-step prediction is insufficient to produce belief states, which is only an issue with approximation error when dealing with empirical results. This is not clearly explained in the paper, but clarified much more nicely in the rebuttal. This would cause me to raise my score from a 3 to a 4 for the misunderstanding, but I still do not find this paper worthy of acceptance. I don't think they are particularly surprising insights, and it seems the sole merit of this paper is an empirical one, and impressive because of performance on complex tasks.

This paper examines the use of generative models for developing representations to improve data efficiency in RL. Specifically, the authors use a generative model that is trained to predict multiple frames into the future (overshooting), and they show that when the model is stochastic (but not deterministic), overshooting leads to useful representations of the environment that can improve RL efficiency. The reviews on this paper were fairly divergent in the first round. Two of the reviewers liked this paper, but one did not feel it provided truly novel contributions, and only brought together previously proposed ideas for using predictive training to improve RL representations. In discussion, the reviewers came to the conclusion that it does demonstrate the utility of overshoot prediction for stochastic models and that an empirical demonstration like this can be useful.

The standard ``serial'' (aka ``singleton'') model of belief contraction models the manner in which an agent's corpus of beliefs responds to the removal of a single item of information. One salient extension of this model introduces the idea of ``parallel'' (aka ``package'' or ``multiple'') change, in which an entire set of items of information are simultaneously removed. Existing research on the latter has largely focussed on single-step parallel contraction: understanding the behaviour of beliefs after a single parallel contraction. It has also focussed on generalisations to the parallel case of serial contraction operations whose characteristic properties are extremely weak. Here we consider how to extend serial contraction operations that obey stronger properties. Potentially more importantly, we also consider the iterated case: the behaviour of beliefs after a sequence of parallel contractions. We propose a general method for extending serial iterated belief change operators to handle parallel change based on an n-ary generalisation of Booth & Chandler's TeamQueue binary order aggregators.

Weaknesses: The absence of any explaining figures makes it hard for the reader to fully understand the underlying architecture of BPNNs. Specifically, it is not clear what the actual target of a BPNN acting on a particular factor graph is! Is it the partition function? Moreover, the intention of the Bethe Free Energy Layer remains obscure. In lines 121-125 you mention: 'When convergence to a fixed point is unnecessary, we can increase the flexibility [...] Additionally we define a Bethe free energy layer using two MLPs that take the trajectories of learned beliefs from each factor and variable as input and output scalars.'

The reviewers felt the paper provides a valuable connection between modern graph neural networks and belief propagation. There was some confusion about how to train the BPNNs, and the paper should be revised to clarify these points. The authors should also expand on discussion about marginal estimation, even if is to highlight limitations of the proposed approach.