Dear authors: your paper was carefully evaluated by the reviewers, and was discussed after we received the rebuttal. There was general agreement that this was an interesting paper and worthy of acceptance at NeurIPS 2019. Adversarial attacks on policy learning in RL is very timely. I would like to note, however, that I solicited some outside feedback on this paper after the reviews were in, and this feedback had both positive and negative comments. This 4th perspective was, I think, particularly on point and worth reading carefully, and I will share it below.

The paper identifies the unsolved problem of meta-Inverse Reinforcement Learning. That is, learning a reward function for an unseen task from a single expert trajectory for that task, using a batch of expert trajectories for different but related tasks as training data (the task being solved by each training expert trajectory is not communicated to the learning algorithm). Because IRL is used rather than imitation learning, a reward function is learned for each task (or rather a single reward function parameterized by the latent variable m which is supposed to capture task). The paper then formulates an framework for training neural networks to solve the identified problem, building off of past work on Adversarial IRL, and adding latent task variables to handle the variation in task. A network q_psi is used to identify the task variable from a demonstration.

The reviewers agree that the paper is interesting and a good contribution.

Clarity: The clarity of this paper is very poor. Value-based is used to mean different things at different points in the paper. This makes the paper very confusing. There is value based to mean value based methods such as Q-learning or SARSA (although no reference to these algorithms or anything like them is made) in the introduction and then value-based to refer to the policy gradient theorem presented in [3]. As discussed in the correctness section, much of the paper is ambiguous and seems wrong or confused in its claims.

Summary and Contributions: This work introduces an algorithm for reinforcement learning in settings with factored action spaces in which each element of the action space may have a different control frequency. To motivate the necessity of such an algorithm, it provides an argument that in this setting, a naive approach with a stationary Markovian policy on the states (which does not observe the timestep) can be suboptimal. Further, it argues that simply augmenting the state or action spaces and applying standard RL methods results in costs which are exponential in L, the least common multiple of the set of action persistences. In constructing the method this paper introduces c-persistent Bellman operators, a way of updating a Q-function in an environment with multiple action persistences, and proves its convergence. This leads to a method which uses L Q-functions, one for each step in the periodic structure of action persistences.

The paper proposes an off-policy policy iteration scheme for factored action spaces where different actions (action dimensions) are persistent with different frequencies. The reviewers agree that the proposed approach is sound, novel, and well motivated. The paper is well written. There is some disagreement how broad the range of applications is to which the proposed method can be applied and what this means for the impact of the paper (R5); some concerns regarding the scalability (R5) of the approach; and some desire for environments not designed by the authors (R2). The AC believes that, although the application domain may be somewhat niche, and the proposed method the result of a somewhat straightforward reasoning about basic properties of MDPs (I don't mean this in a bad way; such basic ideas are often overlooked), on balance the paper will be useful and of interest to the community.

After reading the Author Feedback: The authors addressed and responded to all my concerns in an extensive manner. This is an interesting well-thought contribution, and I am happy to increase my score. Summary: In this paper, the authors investigate how deep reinforcement learning agents with distinct architectures (mainly, feed-forward vs. recurrent) learn to solve an interval timing task analogous to a time reproduction task widely used in the human timing literature, implemented in a virtual psychophysics lab (PsychLab/DeepMind lab). Briefly, in each trial the agent has to measure the time interval between a "ready" and "set" cue, and wait for the same duration before responding by moving their virtual gaze inside a "go" target; with the goal that the duration between the "set" cue and the "go" response should match the interval between "ready" and "set". Time intervals during training are drawn from a discrete uniform distribution.

This paper presents an open source interval reproduction task for RL agents that is based on psychophysics tasks originally developed in neuroscience. This task is used in the paper to better understand how different RL agents solve timing tasks, for example by examining action trajectories and unit activations. This work establishes an open tool and a framework that could be used by future studies to understand computations related to timing in various RL agents. The reviewers all agreed that this paper provides a worthwhile contribution to both the machine learning and neuroscience communities. They had some initial concerns related to generality and scalar variability.

The paper introduces a scalable approach for doing meta inverse RL based on maximum entropy IRL. The baseline is a meta-learning method based on behavioral cloning over which a significant performance improvement is obtained, Pro: The approach seems technically sound, building on the theory of AIRL/GAIL. Also, implementing the equations in a practical and efficient way is a non-trivial contribution. Furthermore, the paper is clearly written. The motivation for IRL versus BC and the advantages that IL can have over RL are clearly explained.

The originality and quality are nice. The main contribution is the introduction of a scalable, meta, inverse RL method, which is quite important and hot topic in IRL. Moreover, the additional experiments from the author response would be good enough to accept. So, I recommend the paper be accepted.