Even after the discussion and the author response there was still some disagreement between the reviewers. The paper proposes a simple yet novel and very interesting idea. There still are a few concerns about clarity, but those can be fixed in the final version (see updated reviews). Overall this is a solid paper, that (as always) would benefit from more thorough empirical evaluation. One reviewer proposed to add an additional baseline of a domain-randomized robust policy that is trained on various tasks.

The paper studies a form of constrained reinforcement learning in which the constraints are bounds on the value functions for auxiliary rewards. This allows a more expressive formulation than the common approach of defining the reward as a linear combination of multiple objectives. The authors show that under certain conditions, the constraint optimization problem has zero duality gap, implying that a solution can be found by solving the dual optimization problem, which is convex. The authors also extend this analysis to the case for which the policy is parameterized. Theorem 1 assumes that Slater's condition holds, which is problematic for two reasons. Slater's condition is usually defined for convex constraints, but the authors specifically state that the constraints in PI are non-convex.

After discussion, the reviewers converge to an accept decision. Please carefully consider the reviewers' comments in revising your paper.

Strengths: - The problem and approach are well motivated and grounded in theory and prior work.

Reviewers are in favor of acceptance after the discussion and I agree. The key novelty in this work is to apply the Multiple Choice Learning framework to model based reinforcement learning. Doing so allows for the model to learn multimodal distributions over future states and the authors provide strong empirical results. Neither dynamics learning nor MCL are novel; however, their combination is novel and demonstrated to be effective. The reviewers have left a number of useful suggestions about how to further strengthen the paper in terms of writing and experimentation and I encourage the authors to make use of this feedback.

Weaknesses: The significance of "breaking the barrier" is somewhat suspicious since it appears to be relevant only when there is a lower bound assumption on the accuracy epsilon, which is a bit strange since we want the accuracy to be high, so the error epsilon to be low. In particular, it doesn't appear to improve on previous results if we make epsilon a constant, for example. EDIT: Thank you to the authors for your response. Here is a bit more explanation of my concern. My comment was inspired by thinking about what are the conditions under which the new bound derived by the authors is actually a strict improvement over the previous bound.

The reviewers appreciated the efforts made by the authors in the rebuttal and updated their reviews accordingly. The reviewers are now all positive about the paper. They are aware that the improvements of the results concern specific regimes for \epsilon, \gamma, but appreciate the results on this fundamental problem. We recommend the paper for acceptance and encourage the authors to account for the reviewers' comments when preparing the camera-ready version of the paper.

The main contribution of this paper is to apply Anderson acceleration to the setting of deep reinforcement learning. The authors first propose a regularized form of Anderson acceleration, and then show how it can be applied to two practical deep RL algorithms: DQN and TD3. Originality: This paper falls under the vein of applying existing techniques to a novel domain. While the idea of introducing Anderson acceleration to the context of RL is not new, as the authors mention, it has not been applied to deep RL methods. While the originality is somewhat limited in this aspect, developing a practical and functional improvement for deep RL algorithms is not trivial.

This work is an interesting contribution to deep RL that considers using Anderson acceleration to improve off-policy TD based algorithms. The approach is supported by some theory as well as experiments on standard benchmark problems. Overall, reviewers like the paper and agree it should be accepted.

Learning value functions is a central theme in reinforcement learning. It is a hard problem because of the non-stationary nature of bootstrapping. This paper proposes a fresh approach for improving the learning of value functions by conditioning them on some information of the future states at training time (hindsight). Conditioning on the right future data should provide more certainty about the future return. All the reviewers liked the premise of the paper, clear motivation, and thorough experiments.