The initial scores were 3478. In the response, authors provide experiment results fusing SOTA trackers on the larger datasets, compared with SOTA trackers, showing improved performance. Authors also provide ablation study using hand-designed rules. During discussion, R2 was satisfied with the new comparisons with SOTA and larger datasets, but was not convinced that the fusion method was useful since there was no ablation study comparing only fusion methods (while keeping trackers the same). R3 was mostly satisfied with the response, but novelty concern was not addressed fully.

Efficient exploration is one of the main challenges in reinforcement learning (RL). Most existing sample-efficient algorithms assume the existence of a single reward function during exploration. In many practical scenarios, however, there is not a single underlying reward function to guide the exploration, for instance, when an agent needs to learn many skills simultaneously, or multiple conflicting objectives need to be balanced. To address these challenges, we propose the \textit{task-agnostic RL} framework: In the exploration phase, the agent first collects trajectories by exploring the MDP without the guidance of a reward function. After exploration, it aims at finding near-optimal policies for N tasks, given the collected trajectories augmented with \textit{sampled rewards} for each task.

Weaknesses: The paper lacks many intricate details that prevents the reader to judge the novelty and full contribution of the work. After reading the rebuttal, an overview of the proposed solution and the problem setting would be of much help to the readers. Is the entire game (with all levels) considered as a POMDP? I see sentences such as "Line 62: environment is considered as a markov process". How is the generalization problem being modelled?

The paper addresses the problem of generalization in POMDPs, and all reviewers agreed that it contains clever ideas which are well evaluated, and so it makes a good contribution. The reviewers also agreed that there are presentation problems that the authors should fix, but that these can be handled in a revision. Hence, I recommend acceptance and very strongly encourage the authors to revise the paper and improve the writing taking into account the detailed comments in the reviews.

SUMMARY: The paper considers the problem of designing a Bellman-like operator with certain properties: 1) Optimality preserving property: The greedy policy of the converged action-value function be the optimal policy. The motivation for the action-gap increasing property comes from the result of Farahmand [12] that shows that the distribution of the action-gap is a factor in the convergence to the optimal policy. Roughly speaking, when the action-gap is large, errors in estimating the action-value function Q becomes less important. The result is that we might converge to the optimal policy even though the estimated action-value function is far from the optimal one. Bellemare et al. [5] propose some operators that have these properties.

The paper proposes a family of robust stochastic operators for RL. This is quite original and potentially impactful. The reviewers raised important questions regarding the clarity of the proofs that was generally answered in the rebuttal. I also read the paper. It makes an important and original contribution.

The introduction of gossip algorithms to Deep-RL is original. The work is generally clearly presented, but some of the reported baseline results do not match previous published works. Figure 1: The IMPALA results look completely off, as do the A3C results on pong, and the A3C results in the appendix. There shouldn't be such a discrepancy between A3C and IMPALA when running with the same hyperparameters (there is a larger discrepancy on some games, but not these ones). I suspect a bug in the IMPALA implementation, or at least an unfair comparison due to all other results using a more recent (and hence more tuned) set of hyperparameters from [Stooke&Abbeel 2018].

The paper introduces gossip based algorithms for consensus and community managing asynchronous updates in distributed Deep RL. The reviewers had some concerns regarding the comparison of the proposed approach to baselines (and the choice thereof), but were overall impressed by the empirical results, that show a more computational efficient algorithm (compared to A2C/A3C) with no significant loss in learning performance. Please take into account the detailed comments of the reviewers (especially R2 and R3) when preparing the final version.

Summary: This paper studies a generalization of online reinforcement learning (in the infinite horizon undiscounted setting with finite state and action space and communicating MDP) where the agent aims at maximizing a certain type of concave function of the rewards (extended to global concave functions in appendix). More precisely, every time an action "a" is played in state "s", the agent receives a vector of rewards V(s,a) (instead of a scalar reward r(s,a)) and tries to maximize a concave function of the empirical average of the vectorial outcomes. This problem is very general and models a wide variety of different settings ranging from multi-objective optimization in MDPs, to maximum entropy exploration and online learning in MDPs with knapsack constraints. In section 2 the authors introduce the necessary background and formalize the notions of "optimal gain" and "regret" in this setting. Defining the "optimal gain" (called the "offline benchmark" in the paper) is not straightforward.

Two out of three reviewers appreciated the contributions of this paper, with one expert reviewer praising almost every aspect of the paper. On the negative side, one reviewer took issue with the proposed setting, highlighting that the utility of the proposed objective function is somewhat dubious in the general context of multi-objective decision making. I agree with this reviewer in that having "multi-objective" in the title of the paper may set the wrong expectations for some readers, and I suggest that the authors consider changing the title of the paper for its final version to avoid such misunderstandings. Furthermore, the final version should discuss the relationship between this paper and the very recent work of Rosenberg and Mansour (2019) that studies essentially the same problem in episodic MDPs. Other than these concerns, the paper is worthy of being published without major changes.