Summary and Contributions: [UPDATE] I have read the rebuttal, and I still believe the authors should work on experiment description clarity. I do not dispute that this paper has committed the common sin of saying "We assume the standard meta-RL framework" and moving on. However, I believe three points are in favour of this paper: - The authors' response seems to indicate that the reviewers' message has been heard and more details are going to be included; I would actually prefer they did not clutter the main paper with these details because ... - The meta-RL methodology for these tasks is very well known and "standard" so if they made changes, it's likely that they made the tasks harder, not easier. There are dozens, perhaps more, papers building on this methodology starting from 2017 onwards, many in top tier conferences, and a majority do not describe the tasks in detail in the main paper. I would still like harder domains, but I can't disregard presented evidence (yet).

This paper was quite controversial among the four reviewers, leading to more than 10 pages of discussion (longer than the paper itself!) In the end, two reviewers were advocating for acceptance (R1, R3), one was advocating for rejection (R2), and one was leaning towards rejection (R4). This is a direction that hasn't been studied before, and will likely become quite relevant in settings where the task distribution is quite heterogeneous The experimental results suggest that the algorithm performs very well on a large number of simple domains, when combined with MAML and RL 2. The experiments also include an ablation study. Time complexity is not an issue; the reviewers appreciated the author response here. These are the main reasons that R1 and R3 were advocating for acceptance. I agree that these are strong points, and make me want to accept the paper.

Weaknesses: Because the idea is new and very interesting, a number of topics can up that could/should be addressed. Is there a way to be certain that the gradient descent using MMFF has the molecule stay on the same basin of the PES that the rigid rotor sampled? It is likely, particularly in crowded conformations that the structure and energy that MMFF reports are not for the same internal angles as the initial torsion angles would suggest. The Gibbs Score is introduced as some completely new idea, but it's essentially related to a (relative) population according to Maxwell Boltzmann statistics. Furthermore, the log of Gibbs score is then a relative free energy, a very intuitive connection with the underlying physics.

The reviewers found this paper to be interesting and compelling, nicely summarized by R2 in discussion: think the method is sound and exciting and the key challenges in transferability live in the availability of (high-accuracy) training data and in the challenges of representation learning for molecules (GCNs need to be exposed to a lot of chemical variability to be able to interpolate in chemical space.). The alkanes are essentially the same bond over and over and lignin is trained and tested in the same chemical space. I insist that these are representation learning challenges to be solved by the community and improvements there could be combined with this RL approach." That said, the reviewers did find several areas where the paper can be improved. Because of space limitations, I understand that not all of these suggestions will be able to be incorporated within page limits, but I do expect the authors will address as much as possible within the main final text, and all feedback addressed either in main text or in a supplementary appendix.

Strengths: 1) This paper focuses on an interesting and practical case of reinforcement learning. Clear examples are provided to demonstrate the difference between predictive knowledge (general RL) and retrospective knowledge (this work), how RL with retrospective knowledge can be used in real-world applications, and why general RL algorithms (GVFs) fail to represent such knowledge. The formulation is general so that multiple existing RL algorithms can be extended to the Reverse RL setting. Theoretical analysis is given to justify the convergence of Reverse RL algorithms (under linear function approximation).

Additional Feedback: Response to author feedback: From the informal discussion about the cross-component counters, I'm getting that it's somehow bad if different components have been explored unevenly and therefore encouraging more balanced exploration (pairwise) reduces overall variance in the amount of exploration between components. I'm sure there's a lot I'm not getting, but that helps a bit. I think it should be the case that you recover an object when you multiply its factors together (for the appropriate definition of "multiply"). There are papers (well, just one I can think of) that deal with truly factored MDPs that are the product of simpler MDPs. They correctly call their MDPs factored.

While this paper initially had some mild divergence of opinion among the reviewers, after the author response and some detailed discussion, it was agreed that this paper makes a solid contribution (please see the revised reviews). It is certainly is of relevance to NeuRIPS. After discussion, there was agreement on the significance of the conceptual contribution, namely the treatment of the cross-component bonuses. Several reviewers note that the mathematics is fairly "standard" (Bernstein-bound machinery), though in the end that should not be considered a drawback. At least one reviewer notes that the 31pp appendix means that it is not possible to verify the mathematical results during the review period.

The paper investigates various data augmentation techniques in the context of RL, and shows that they lead to improved performance. The method is simple and can be applied to different RL algorithms. One may argue that the algorithmic contribution is not majorly novel, but the simplicity of the method and the improvement in the performance, as well as the unanimous favourable opinion of the reviewers, would be good enough justification to recommend acceptance of this work. I encourage the authors to consider the comments of the reviewers in revising their paper. I would also like to ask the authors to increase the number of runs/seeds in some of their experiments from 3 or 4 to a larger number (10).

Summary and Contributions: The paper proposes a method for utilizing ODEs to represent dynamics for continuous-time decision-making problems with the aim of They also target filling a perceived gap in the literature of Deep RL for continuous-time problems, where most publications are model-free and discretize time if it is continuous. They claim that their approach leads to lower dependence on vast amounts of training data, better performance and that the model-based approach is well-founded. I tend to agree, although this is not exactly my area. I also believe the importance of connecting ODEs and other explicit models is critical for extending RL methods to important problems in physics, chemistry, epidemiology and population modelling.

The reviewers all found the paper to make a reasonable contribution. While there are some limitations that were pointed out, the rebuttal addressed them well and the experiments were appreciated.