The paper proposes an unifying view on several interesting problems for the RL community (reward maximization, pure-exploration, risk averse RL). It presents a generic Policy Gradient Theorem and studies the convergence of the corresponding policy gradient ascent, which is an important contribution.

This paper formalizes the problem of inverse reinforcement learning in which the learner's goal is not only to imitate the teacher's demonstration, but also to satisfy her own preferences and constraints. It analyzes the suboptimality of learner-agnostic teaching, where the teacher gives demonstrations without considering the learner's preferences. It then proposes a learner-aware teaching algorithm, where the teacher selects demonstrations while accounting for the learner's preferences. It considers different types of learner models with hard or soft preference constraints. It also develops learner-aware teaching methods for both cases where the teacher has full knowledge of the learner's constraints or does not know it.

The paper proposes a really interesting and novel variant of inverse RL with a nice formalization. The proposed algorithms are suitable. While the reviewers felt that the empirical results were weak (lack of scalability and linear reward function limitation), they thought that this was outweighed by the novelty of the problem and the significance of the contribution.

I think that reporting it in the introduction is premature, I suggest to describe the meaning of the theorem without the statement.

The paper proposes an adaptation of the classical Q-learning algorithm with linear function approximation that enjoys polynomial sample complexity. All reviewers feel the paper contains interesting contribution to the RL literature that should appear in this conference, and I therefore recommend acceptance.

This paper studies the problem of learning useful representations for reinforcement learning through the lens of an adversarial framework. In particular, a good representation is identified as one that yields low linear value-function estimation error if an adversary is able to choose a value function (induced by a policy). The paper shows first that the the only policies that should be considered are deterministic, and then identifies a more narrowed set of adversarial values, though the number is still exponential. I really liked the theoretical insights of this paper, and because of this I tend to vote for acceptance, though I claim that experiments are too preliminary. Some more comments below: 1- in (1) highlight more clearly that \phi is the only optimization knob.

This provides a new conceptual/theoretical understanding of representation learning for RL that all reviewers felt was interesting. The experimental results are somewhat preliminary, but sufficient.

This paper proposes a new multi-task hierarchical reinforcement learning algorithm. The high-level policy achieves the assignment of tasks by solving a linear programming problem(or a quadratic programming problem), and the low-level policy is pre-defined. The biggest contribution of this paper is to get rid of the limitation of the number of agents and the number of tasks by modeling the multi-task assignment problem as an optimization problem, which based on the correlation between the agent and the task and the correlation between the tasks. After training the correlation in a simple task, you only need to re-solve the optimization problem in the complex task, without retraining, thus achieving zero-shot generalization. In this paper, the collaboration patterns between agents in the multi-task problem, such as creating subgroups of agents or spreading agents across tasks at the same time, are transformed into constraints to be added to the optimization problem corresponding to the high-level policy.

Additional Feedback: The authors' response has addressed my questions. I will keep my score. This is a natural question to ask, so it could be worth an explanation somewhere. However, this paper suggests a slower rate by a factor of (1-\gamma) {-2}. What could cause the difference and how could the theory here guide development of deep RL algorithms?

Additional Feedback: After Authors' Reponse: I still find the paper's analysis regarding action-gaps a bit weak, and the authors' response didn't help much in that regard. I think their action-gap analysis needs to be considered under the new findings of (van Seijen et al., 2019); increasing the action-gap is not important on its own, rather it's the homogeneity of the action-gaps across the states that is important. While I still stand by my verdict of accepting this paper, in light of other reviews, I think the paper's writing should be toned down a bit in regards to its theoretical novelty and claims about empirical results (e.g. the first non-dist-RL to beat a dist-RL). Q1: To the best of my knowledge, IQN in Dopamine also uses Double Q-learning. Is this also the case for your M-IQN agent?