The paper proposes an improvement of popular'successor representation' approaches in reinforcement learning via a mechanism for maintaining and quickly updating a distribution over multiple successor maps. This innovation enables the model to adapt better to environmental changes such as different goals or reward structures. All three reviewers agree that this is a strong paper that should be accepted. I see no reason to contradict their opinion. While the reviewers were very positive, they did point out issues of clarity in the exposition, and we would like to remind the authors that their paper will reach a wider audience if they can make the presentation and explanation as clear and simple as possible in the camera ready version.

As such, it opens up potential new research approaches along with providing an improvement on the SOTA. Quality: The argument is well-developed, and extensive proofs are provided in the supplementary materials or referenced in existing literature. The greedy approach is directly applied to two existing SOTA full-planning-based algorithms, suggesting it is a generalizable alternative. Clarity: The paper is generally well-organized and clear; the paper gives an intuitive sense of the results, although the bulk of the proofs are confined to the supplementary material. Several scattered clarity issues are described in the detailed comments below.

All reviews agree that the contribution is novel and strong. The rebuttal gave important answers and we all strongly defend acceptance.

Post Response update: Thank you for the detailed response. I still believe that a more in depth discussion of the differences or similarities of policy and cumulant based formulations is required to place the paper appropriately in context of prior work. I think the new results presented by the authors in the response partially address my concerns about comparisons with prior work but not fully. I would still like to see comparison against a policy-based method as per the authors' classification. I agree that all methods might have negative transfer but it would be ideal to include a discussion of the conditions under which the methods would show positive or negative transfer (something that the authors do) and to place that in context with other methods at least qualitatively (something that the authors dont). The newer evaluations in the response do satisfy a part of my concerns.

Originality: The authors apply the idea that overparametrization induces local linearization, which has been documented for supervised learning, and in another submission for TD learning. In particular, they decompose the error into two terms, one due to TD, and the other due to SGD, and incorporate them in the analysis of infinite-dimensional mirror descent. The insight that the previous previous analysis for TD could be generalised to a meta algorithm that includes both TD and SGD as particular cases is key. Related work is adequately cited, and differences with previous works are clearly stated, including differences with the sister submission [5]. Quality: The submission seems technically sound, and includes detailed proofs (I just skimmed through them). This is a complete piece of work.

Summary and Contributions: This paper proposes a framework for goal-conditioned RL with a goal representation whose structure is learned from weak human supervision. Most goal-conditioned RL methods either use the raw image as a goal, or an encoding learned with an unsupervised method such as a VAE. This paper takes as input a (relatively small) dataset of images, and asks human annotators to rank semantic attributes for pairs of image (which has higher lighting, which one has a door which is more open, etc). The algorithm operates in two phases: 1. Using the weak supervision signal from the human annotators, a disentangled representation is learning using a GAN-type loss on triplets of 2 images and one binary label.

The paper proposes a way to incorporate weak supervision, in the form of pairwise comparisons along various axes, into a goal-directed reinforcement learning framework, showing how this supervision can identify relevant latent factors for the construction of new tasks. The reviewers agree that this is a novel approach and makes an important step toward fully unsupervised approaches. As such, we are recommending acceptance.

Weaknesses: I have several concerns about the proposed approach. First, the empirical results give mixed messages. In one out of three tasks (i.e., reacher), the LfL baseline significantly outperforms LOGEL (Figure 4, left). Whereas for another task (i.e., hopper), the policy trained with the reward function recovered by LOGEL outperforms the policy trained on the true reward function. And what kind of reward function does the LfL baseline recover for the hopper task, that leads to no learning at all?

Drawing upon Inverse RL, the submission proposes learning from an expert, which is using a learning process to optimize its reward. In the initial reviews, three of four reviewers were positive on the submission, and after seeing the author feedback, one of the reviewers was persuaded to raise the overall score, so that the current scores are now (7, 7, 6, 5). With these scores, it will be likely (but not guaranteed) to be accepted to NeurIPS. Regardless, it is important to, and we trust that you will, address all of the issues that were raised by the reviewers in the next version of the manuscript.

The paper addresses the issue of exploiting correlation structures in Markov Decision Processes with discrete state spaces. The authors identify a gap that currently makes working with discrete state spaces problematic - that there is no principled method for modelling the state correlations that is flexible enough to accommodate all the ways in which these correlations could be exploited. The paper presents a hierarchical Bayesian model and proposes a variational inference method to find solutions. The model and procedure presented in the paper are an original application of variational inference, and represent a more general method for dealing with correlation structures than anything I have encountered before. The authors have done a great job of demonstrating this by employing three vastly different problem domains. It is unusual to see Imitation Learning, System Identification and Reinforcement Learning all being tested under a new model in one paper.