Weaknesses: I also feel that the paper could have benefited from a discussion of these as compared to just outrightly saying that existing methods do not give us good results. In particular, the conditions under which existing methods work vs do not work should have been discussed more explicitly than what it is right now in the paper. Moreover, I think the experiments on cartpole and hopper are not indicative of their method's performance since these have determnisitc dynamics and the dataset was collected as trajectories (so s' is as frequent as s in the distribution \mu, see my point below) and hence their choice of masking reduces to action conditioned masking only. Some other questions that I have: - From the analysis perspective, the paper says that prior works such as Kumar et al. 2019 that use action conditional and concentrability do not get the same error rate. Is the main issue behind this limitation that the notion of concentrability used in Kumar et al. and other works is trajectory centric and not on the state-action marginal?

This is a nice paper, with a new idea and strong theoretical backing. The reviews, rebuttal and discussion periods led to a lot of detailed feedback, so I'd encourage the authors to include as much of this as possible in the camera-ready version, and specifically revise for clarity around the points that were unclear to the reviewers in the first submission.

The theoretical claims of producing lower bounds on Q values are not sufficient since there is no proof that the conservative Q values are anywhere near the true Q values. Just estimating Q 0 for positive rewards could give the same result at theorems 3.1, 3.2, and 3.3. Clearly, the algorithm is doing something smarter than this, but the current analysis does not characterize what the algorithm is doing. The gap expanding result is likely the strongest of the four theorems, but without doing the work to connect this back to why this will actually help performance it is still difficult to judge. Moreover, no comparison is made to the overestimation that would happen without the proposed algorithmic change.

All the reviewers were positively impressed by the rebuttal provided by the authors, which clarified many of their concerns. Their updated scores and the final decision to propose acceptance for the paper are based on the requirement that the authors will significantly change the paper integrating the insights presented in the rebuttal as well as clarifications of the few remaining points that have not be addressed yet (see R3).

I believe the proposed method, HAL (Hierarchical Abstraction with Language), is an interesting approach for HRL. The authors adapt Hindsight Experience Replay for instructions (called Hindsight Instruction Relabelling). I have some concerns about the experimental setup and empirical evaluation of the proposed method: - The motivation behind introducing a new environment is unclear. There are a lot of similar existing environments such as crafting environment used by [1], compositional and relational navigation environment in [2]. Introducing a new environment (unless its necessary) hinders proper comparison and benchmarking.

The additional experiments presented in the abstract addressed many of the reviewers concerns; however, there was some doubt that these changes will be successfully incorporated into a camera ready. These additions (especially the use of a full language model in the policy and the crafting world results) would significantly strengthen the paper and I strongly urge the authors to follow through on their rebuttal commitment of integrating these results in future revisions. There are also concerns that the approach is highly specialized for the environment and is limited by its need for automatic goal language prediction / verification to perform HIL. Given the content of the paper already, this might be better left to future work.

Additional Feedback: Page 2: In your related work, you have missed several important works, such as for example those of Francis Maes where he proposes approaches for learning fundamental learning rules for RL algorithms (especially for playing bandit problems), see https://scholar.google.be/citations?hl fr&user h8kelPwAAAAJ His approach is very close to yours (same type of objective function). Page 3: The finding of an optimal update policy is in some sense expressed as a Bayesian RL problem (you know a probability distribution over environments as prior) but you never make the connection with this field of research. In the work of Maes, it is somehow formalized as such. You approach can be considered as a gradient-based direct policy search approach for which you have as evaluation metric formula (1), as search space \eta \times \theta and as optimization method a gradient-based method. The main contribution of this paper is how to define the candidate space of your eta, something you never define very well.

The third recommended rejection, but did not argue for rejection in the discussion. Despite the overall positive response, the reviewers shared R1's concerns about missing related work.

Overall, the method provided is a straightforward application of a known IR method to MARL, the results are promising and the writing is clear. As such, this work has limited novelty but provides good empirical contributions, though these too could be improved by considering more domains. A more detailed review of the paper, along with feedback and clarifications required are provided below. The work is motivated by the claim that providing individual IRs to different agents in a population (in a MARL setting) will allow diverse behaviours. The analysis at the end of the paper shows that a lot of the learned IR curves do overlap.

The paper extends the idea of learning intrinsic rewards to the centralized learning - decentralized execution, cooperative multi-agent setting. This setting has become popular in past years, as a setting that has high potential for real world applications and being amenable to progress towards tractable solutions. The approach presented by this work is easy to conceptually simple and well motivated. The authors empirically show that it outperforms existing state of the art approaches on challenging StarCraft benchmark tasks. Reviewers raised several concerns about the paper, including clarity (experiment details, precise description of the approach and distinction from existing approaches), and the need for further analysis.