The paper investigates the problem of inferring an agent's belief of the system dynamics of an MDP, given demonstrations of its behavior and the reward function it was optimizing. Knowledge of this internal belief can be used for Inverse Reinforcement Learning of an unknown task in the same environment. Furthermore, given the action provided by the agent, its intended action on the true dynamics can be inferred. This allows for assistive tele-operation, by applying the intended actions to the system instead of the provided ones. The proposed method models the agent using the model derived in maximum causal entropy inverse reinforcement learning.

Summary The authors present a reinforcement learning technique based on importance sampling. A theoretical analysis is performed that shows how the importance sampling approach affect the upper bound of the expected performance of the target policy using samples from a behavioral policy. The authors propose a surrogate objective function that explicitly mitigates the variance of the policy update due to IS. Two algorithms are proposed based on natural gradients for control-based (learning low-level policy) and parameter-based problems (discrete low-level controller with stochastic upper-level policy). The algorithms were tested on standard control tasks and are compared to state of the art methods.

In this paper, the authors proposed a hierarchical visual architecture that operates on a saliency map and uses a novel attention mechanism based on 2D Gaussian model. Furthermore this mechanism sequentially focuses on salient regions and takes additional glimpses within those regions in multi-label image classification. This sequential attention model also supports multiset prediction, where a reinforcement learning based training procedure allows classification to be done on instances with arbitrary label permutation and multiple instances per label. Pros: 1) This paper proposes a novel saliency based attention mechanism that utilizes saliency in the top layer (meta-controller) with a new 2D Gaussian based attention map. This new attention map models the regional /positional 2D information with a mixture of Gaussian distributions, which is more general than the standard attention layer (in DRAW, Show-attend-tell), where attention is enforced based on softmax activation. This mechanism is intuitive as it's inspired by human-level attention mechanism.

This paper addresses the challenge of an environment with discrete, but large number, of actions, by eliminating the actions that are never taken in a particular state. To do so, the paper proposes AE-DQN which augments DQN with contextual multi-armed bandit to identify actions that should be eliminated. Evaluation conducted on a text-based game, Zork, shows promising results, as AE-DQN outperforms baseline DQN on several examples. This idea of eliminating actions which are never taken in a given state is a sound on. The paper is clear and well written.

The paper defines a new type of reinforcement learning algorithm, which takes account of the imperfections of the function approximator and tries to obtain the best policy available given these imperfections rather than assuming no imperfections exist, thus avoiding pathologies arising when we assume a flawed approximate is perfect. The quality of this paper is really good. It introduces a new type of RL algorithm, which is clearly motivated and solid. The weaker points are: 1. The complexity of the defined algorithm seems too high for it to be immediately applicable to interesting problems.

The paper "Learning Temporal Point Processes via Reinforcement Learning" proposes a new way to lean temporal point processes, where the intensity function is defined via recurrent neural networks rather than classical parametric forms. This enables a better fitting with the true generative process. It builds on the WGAN approach but rather than dealing with a minimax optimization problem, the authors propose to use a RKHS formalization to find an analytic maximum for the expected cumulative discrepancy between processes underlying observed sequences and generated ones. The results look convincing and the followed inverse reinforcement learning approach elegant, but I am missing some justification and clarifications w.r.t. Authors claim that their way of learning allow them to define better specified models than parametric based ones.

Summary: This paper studies RL exploration based on uncertainty. First, they compare several previously published RL exploration methods and identifying their drawbacks (including illustrative toy experiments). Then, they extend a particular previous method, bootstrapped DQN [1] (which uses bootstrap uncertainty estimates), through the addition of random prior functions. This extension is motivated from Bayesian linear regression, and transferred to the case of deep non-linear neural networks. Experimental results on the Chain, CartPole swing-up and Montezuma Revenge show improved performance over a previous baseline, the bootstrapped DQN method.

The article extends previous work of primal-dual optimisation for policy evaluation in RL to the distributed policy evaluation setting, maintaining attractive convergence rates for the extended algorithm. Overall, the article gradually builds its contribution and is reasonably easy to follow. A few exception to this are the start of related work, dropping citations in lists, and the lack of an explanation of the repeatedly mentioned'convex-concave saddle-point problem'. The authors equate averaging over'agents' with averaging over'space', which is somewhat of an imprecise metaphorical stretch in my view. The contribution is honestly delineated (collaborative distributed policy evaluation with local rewards), and relevant related work is cited clearly.

This paper proposes a Bayesian model for adversarial learning problem. Empirical studies on Fashion-MINST and traffic sign recognition show that the proposed methods is slightly better than other adversarial learning baselines. Below I list my concerns about the paper: For modeling, 1. This paper ignore a highly relevant work'Bayesian GAN' [1]. The non-cooperative game between'data generator' and'learner' established in this paper is almost the same as the vanilla GAN.

This paper presents a theorem proving algorithm that leverages Monte-Carlo simulations guided by reinforcement learning from previous proof searches. In particular, to guid the search, the authors use the UCT formula augmented by (1) learning prior probabilities of actions for certain proof states (policy learning), and (2) learning the values corresponding to the same proof states (policy evaluation). The proposed algorithm is using no domain engineering. The authors evaluate their proposal on two datasets: Miz40 and M2k. The evaluation shows the proposed solution outperforms state-of-the are by solving 40% more problems.