The paper presents and evaluates SPECTRL, a framework for transforming formal specifications of tasks into shaped reward function. The reviewers agreed that, while it is not obvious that this paper will be extremely impactful, it is nonetheless interesting, convincing, and clearly written. After some discussion, the consensus leans towards acceptance, although with some outstanding issues (especially regarding the cartpole results) which should be addressed before publication. It is also highly recommended that a reference implementation of this method be released for use within the community, although it is not in my power to make this a formal requirement for publication.

POST-REBUTTAL I thank the authors for their detailed response. My main concern was the level of experimental detail provided in the submission, and I'm pleased that the authors have committed to including more of the details implicitly contained within the code in the paper itself. My overall recommendation remains the same; I think the paper should be published, and the strong Atari results will be of interest fairly widely. However, there were a few parts of the response I wasn't convinced by: (1) "(D) Inefficient Hyperparameter": I don't agree with the authors' claim that e.g. QR-DQN requires more hyperparameters than FQF (it seems to me that both algorithmically require the number of quantiles, and the standard hyperparameters associated with network architecture and training beyond that).

The reviewers expressed some concerns about the significance of the paper, given that the main contribution is a SOTA result. However, they conclude that the Atari benchmark is sufficiently mature that an increase in this direction is of general interest. Some of the sticking points that should be addressed in the revision are: 1) consider performing additional empirical analysis to better understand how the method operates, 2) include further details (as requested by the reviewers).

Additional Feedback: Overall, the paper is quite well-written and the motivation and idea are simple and interesting. Except for the two main concerns that I will describe below, I'm mostly satisfied with the quality of this paper; thus, I'm willing to increase my score if the authors can address the following concerns. Both of these ideas have been proposed and used in several previous HRL works; thus, this can be seen as a combination of two existing ideas. This idea has been already used in [1-4], even though some of these works in different settings. For example, they predict the "distance" between current state and the sub-goal state (e.g., UVF with -1 step reward [1, 2] or success rate of (random) low-level policy [3], or k-step reachability [4]), and 2) limit the sub-goal generation to choose the near-by subgoals only (e.g., thresholding [1-4]).

The paper addresses exploration in actor critic methods where the authors identify 2 main problems: pessimistic under-exploration and Directional uninformedness. The authors propose to use UCB upper and lower bounds based on the uncertainty of the value function. All reviewers appreciated the intuitive idea and the exhaustive evaluation of the approach. The results were also considered to be very promising and the authors provided additional ablation studies with their rebuttal. There was a consensus of all reviewers that the paper is a valueable contribution to the field of reinforcement learning.

The baselines are somehow weak. Though TorchBeast is a strong baseline, the PPO and A2C from Kostrikov seem weak. As far as I know, faster training is not the goal of Kostrikov's implementation. For PPO, the implementation from OpenAI baselines are stronger, which features parallelization with MPI and all-reduce gradients. For A2C, one could consider rlpyt (rlpyt: A Research Code Base for Deep Reinforcement Learning in PyTorch), where various sampling schemes (including batch synchronization) and optimization schemes can be used.

This paper proposes a synchronous training scheme for reinforcement learning which address issues with existing synchronous methods (low throughput) and existing asynchronous methods (unstable, non-reproducible, etc.). The reviewers viewed this more of an engineering paper, but the design, execution, and experiments are solid, so we are recommending acceptance. I saw that the paper mentions that code will be released, but I want to emphasize the importance of this, as a large part of the value here is in enabling others to build on and use the proposed method.

According to the papre, ReBeL is a novel method to deal with two-player zero-sum imperfect-information games. And it may be able to be used to solve other inperfect-information problem. And the domain of this paper, multi-agents RL in imperfect-information, has high relevance to NIPS. 6. The experiment compared the module with human player, which is a strong evidence of the exploitability of ReBeL.

Weaknesses: (This section is being combined with "comments for improvement" section below) A bit of a nitpick regarding the language use around "more" or "less" resources. The authors write about an agent using "more resources", which corresponds to "lower entropy" for the actions in a particular state. I think, though, that technically (and literally, for this agent) the amount of resources used for each memory is exactly the same; it's literally a number to represent the mean and standard deviation. From what I can tell, the authors are arguing that memories with lower standard deviations would *require* more resources to represent in certain implementations (such as in brains). So it's not actually the case that more resources are used for low-sigma memories in the agent, but that more resources might be used in other agents.

This paper nicely bridges between neuroscience and RL, and considers the important topic of limited memory resources in RL agents. The topic is well-suited for NeurIPS (R2) as it has broader applicability toward e.g. All reviewers agreed that it is well-motivated and written (R1, R2, R3, R4), although R3 did ask for a bit more explanation on some methodological details. It is also appropriately situated with respect to related work (R1, R2, R3) although R2 suggests a separate related works section, and R4 wanted to see more discussion of work outside of neuroscience, focused on optimizing RL with limited capacity. R1 pointed out that perhaps there's a bit of confusion between memory precision and use of memory resources, as the former is more accurate for agents, the latter perhaps for real brains - ie more precise representations require more resources to encode in the brain, but this seems to be a minor point.