Weaknesses: - One weakness of the successor representation is that it is policy-dependent. So, in the control setting, it would need to be relearned whenever the policy is modified. On the other hand, perhaps one-step models would not suffer from this problem (since they are conditioned on actions too). Could you comment on this issue? So, it would seem like, when the model outputs a prediction, the agent would not know how far into the future this state is---it could be the very next state or far into the future.

Summary: this paper proposes a new model-based RL algorithm, where instead of learning state transition probabilities, the occupancy distribution for an infinite horizon is learned. This method can be seen as an extension of the method known as the successor representation to continuous state-action spaces and to infinite horizons. The occupancy distribution is modeled as an energy function, and learned with temporal differences (TD), using a GAN. The experiments on a few MuJuCo problems clearly show the advantages of the proposed approach compared to RL algorithms such as PPO and SAC. The reviewers agree that the proposed method is new, interesting, and validated by the simulation experiments.

Learning a near optimal policy in a partially observable system remains an elusive challenge in contemporary reinforcement learning. In this work, we consider episodic reinforcement learning in a reward-mixing Markov decision process (MDP). There, a reward function is drawn from one of M possible reward models at the beginning of every episode, but the identity of the chosen reward model is not revealed to the agent. Hence, the latent state space, for which the dynamics are Markovian, is not given to the agent. We study the problem of learning a near optimal policy for two reward-mixing MDPs.

Additional Feedback: Global comments: I think the authors did not use the proper NeurIPS LaTex template. Line numbers are missing, as well as footnotes. The global look and number of pages seem ok though. Figures 1, 2 and 3 are of poor quality, as they appear visually pixelated. I strongly suggest the authors to use images in vector graphics.

The reviewers all agree that the paper is above the acceptance threshold. There is novelty in how the paper applies learning to JSSP, and the results are promising. But as Reviewer 4 points out, the paper hasn't shown how the proposed approach trades off solution quality and running time, without which it is difficult to judge whether it is a significant advance over existing techniques. Adding such results will strengthen the paper considerably.

This work is original in its use of deep reinforcement learning and graph neural networks to learn novel search control heuristics for SAT solving. While the techniques used are not novel themselves, the application domain is. The authors do a good job of surveying related work in this area and situating their contributions in this landscape. The paper is well-written and I found it very easy to follow the details of the proposed approach and the authors' results. Technically, the work presented is solid, though I have a few comments/suggestions here.

Weaknesses: While I like the duality result, I find this paper is not substantial enough that merits acceptance. This paper shows a class of model-optimistic algorithms can be implemented efficiently (with minor modifications). However, none of state-of-the-art algorithms is model-optimistic algorithms. This is somehow inherent with this class of algorithms because the transition model scales with S 2 but the optimal bounds scale linearly in S via value-optimistic algorithms. Value-optimisic algorithms are not only more computationally efficient but also more statistically efficient. So making model-optimistic algorithms more efficient is not a very significant result.

The reviewers are in agreement that this is interesting and well-presented work. The main concern was about the extent to which the results will help us derive SOTA algorithms in the future. I find the contribution reasonable without this and hope the community will figure out how/if these results are useful. Please do take the reviewers minor suggestions into consideration when preparing a final version.

I think the results are much more comprehensive now. I raised my score accordingly. If I understand the main idea correctly, the proposed method can be interpreted as a gradient-based meta-learning method (e.g., MAML) in that the algorithm finds the gradient of the main objective by taking into account the parameter update procedure. It would be good to provide this perspective and also review the relevant work on meta-gradients for RL (e.g., MAML [Finn et al.], Meta-gradient RL [Xu et al.], Learning intrinsic reward [Zheng et al.]). Nevertheless, I think this is a novel application of meta-gradient for tuning auxiliary task weights.

Reviewers agreed that the paper addresses an important problem in current deep RL research and appreciated the effort put into the rebuttal by the authors. New experiments using the 0/1 reward formulation and a comparison to fixed hand-tuned hyper parameters addressed two of the main concerns raised by reviewers. In the end all three reviewers recommended accepting the paper.