The authors find that in certain environments, the learned strategies are consistent with predictions from economics.

In this paper, the authors studied the problem of multitask reinforcement learning (MTRL), and propose several optimization techniques to alleviate the scalability issues observed in other methods, especially when the number of tasks or trajectories is large. Specifically, they rely on consensus algorithms to scale up MTRL algorithms and avoid the issues that exist in centralized solution methods. Furthermore, they show how MTRL algorithms can be improved over state-of-the-art benchmarks by considering the problem from a variational inference perspective, and then propose a novel distributed solver for MTRL with quadratic convergence guarantees. In general, this work is tackling some important problems in the increasingly popular domain of multi-task RL. Using the variational perspective of RL, the problem of MTRL can be cast as a variational inference problem, and policy search can be done through the minimization of the ELBO loss. To alternate the updates on variational parameters and the policy parameters, the authors also propose using EM based approaches, which is very reasonable.

This paper proposes an algorithm for learning goal-conditioned RL policy, in which a goal is defined as a single image. The authors propose to encode a state (an image) to a vector in latent space using variational autoencoder, and define reward functions inside the latent space. The paper shows that such reward function outperforms baseline such as pixel based reward functions. The authors then proposed latent goal relabeling, which generates new goals and rewards given an exist tuple (s, a, s'). Finally, the authors propose goal imagination, which samples goals from latent space during training, essentially allowing training without specifying a particular goal.

Review of submission 1489: EX2: Exploration with Exemplar Models for Deep Reinforcement Learning Summary: A discriminative novelty detection algorithm is proposed to improve exploration for policy gradient based reinforcement learning algorithms. The implicitly-estimated density by the discriminative novelty detection of a state is then used to produce a reward bonus added to the original reward for down-stream policy optimization algorithms (TRPO). Two techniques are discussed to improve the computation efficiency. Comments - One motivation of the paper is to utilize implicit density estimation to approximate classic count based exploration. The discriminative novelty detection only maintains a density estimation over the states, but not state-action pairs.

The paper defines "Uniform-PAC" where uniformity is over the optimality criterion, eps. It is PAC like in that optimal actions are taken in all but a bounded number of steps. It is also regret like in that the algorithm is eventually good relative to any epsilon---not just one it is told to meet. I thought the discussion of different performance metrics was thorough and informative. I would have liked more intuition about the iterated logarithm idea and its main properties, but I understand that the highly technical stuff had to be expressed in very limited space.

The main idea is to demonstrate the effectiveness of these simple algorithms compared to the much more complex state-of-the-art RL algorithms proposed and evaluated on MuJoCo tasks. The results of the empirical evaluation are startling. The paper convincingly demonstrates very strong performance of the simple algorithm and policy class on the MuJoCo tasks. The evaluation is extremely thorough, the results are compelling and raise serious questions about the current state of RL algorithm evaluation methodology using MuJoCo. In my opinion, this paper is an excellent contribution to the RL literature.

These specifications can be seen as non-Markovian reward functions. Thus, this work is related to inverse reinforcement learning (IRL) which aims to infer the reward function of an agent by observing these successive states and actions. By defining the probability of a trajectory knowing a specification (using the maximum entropy principle) the development leads to a posterior distribution. Two algorithms result from this and allow to test the approach on the system presented in introduction (motivating the paper).

R5: Summary: This paper builds on the basic idea of the Horde architecture: learning many value functions in parallel with off-policy reinforcement learning. This paper shows that learning many value functions in parallel improves the performance on a single main task. The novelty here lies in a particular strategy for generating many different reward functions and how to combine them to generate behavior. The results show large improvements in performance in an illustrative grid world and Miss Pac-man. Decision: This paper is difficult to access.

The paper presents an approach to performing transfer between multiple reinforcement learning tasks by regularizing the policies of different tasks towards a central policy, and also encouraging exploration in these policies. The approach relies on KL-divergence regularization. The idea is straightforward and well explained. There are no theoretical results regarding the learning speed or quality of the policies obtained (though these are soft, so clearly there would be some performance loss compared to optimal). The evaluation shows slightly better results that A3C baselines in both some simple mazes and deep net learning tasks. While the paper is well written, and the results are generally positive, the performance improvements are modest.

The paper "Deep Reinforcement Learning of Marked Temporal Point Processes" proposes a new deep neural architecture for reinforcement learning in situations where actions are taken and feedbacks are received in asynchronous continous time. This is the main novelty of the work: dealing with non discrete times and actions and feedbacks living in independent timelines. I like the proposed architecture and I think the idea can be of interest for the community. However, from my point of view several key points are missing from the paper to well understand the approach and its justification, and also for a researcher which would like to re-implement it: - For me, it would be very important to discuss more about marked temporal process. Why is it better to model time like this rather than using for instance an exponential law?