Node-perturbation learning is a type of statistical gradient descent algorithm that can be applied to problems where the objective function is not explicitly formulated, including reinforcement learning. It estimates the gradient of an objective function by using the change in the object function in response to the perturbation. The value of the objective function for an unperturbed output is called a baseline. Cho et al. proposed node-perturbation learning with a noisy baseline. In this paper, we report on building the statistical mechanics of Cho's model and on deriving coupled differential equations of order parameters that depict learning dynamics. We also show how to derive the generalization error by solving the differential equations of order parameters. On the basis of the results, we show that Cho's results are also apply in general cases and show some general performances of Cho's model.

In this paper we combine one method for hierarchical reinforcement learning - the options framework - with deep Q-networks (DQNs) through the use of different "option heads" on the policy network, and a supervisory network for choosing between the different options. We utilise our setup to investigate the effects of architectural constraints in subtasks with positive and negative transfer, across a range of network capacities. We empirically show that our augmented DQN has lower sample complexity when simultaneously learning subtasks with negative transfer, without degrading performance when learning subtasks with positive transfer.

Designing effective exploration-exploitation algorithms in Markov decision processes (MDPs) with large state-action spaces is the main challenge in reinforcement learning (RL). In fact, the learning performance degrades with the number of states and actions in the MDP. However, MDPs often exhibit a low-dimensional latent structure in practice, where a small hidden state is observable through a possibly large number of observations. In this paper, we study the setting of rich-observation Markov decision processes (\richmdp), where hidden states are mapped to observations through an injective mapping, so that an observation can be generated by only one hidden state. While this mapping is unknown a priori, we introduce a spectral decomposition method that consistently estimates how observations are clustered in the hidden states. The estimated clustering is then integrated into an optimistic algorithm for RL (UCRL), which operates on the smaller clustered space. The resulting algorithm proceeds through phases and we show that its per-step regret (i.e., the difference in cumulative reward between the algorithm and the optimal policy) decreases as more observations are clustered together and finally, matches the (ideal) performance of an RL algorithm running directly on the hidden MDP.

This paper introduces Dex, a reinforcement learning environment toolkit specialized for training and evaluation of continual learning methods as well as general reinforcement learning problems. We also present the novel continual learning method of incremental learning, where a challenging environment is solved using optimal weight initialization learned from first solving a similar easier environment. We show that incremental learning can produce vastly superior results than standard methods by providing a strong baseline method across ten Dex environments. We finally develop a saliency method for qualitative analysis of reinforcement learning, which shows the impact incremental learning has on network attention.

Facebook fighting extremism with AI -- "The problem, as usual, is determining what is extremist, and what isn't, and it goes further than just jihadists," he said. "Are they just talking about ISIS and Al Qaeda, or are they going to go further to deal with white nationalism and neo-Nazi movements?" AI is big business -- Element AI raises a whopping $102 million to bridge the gap between the haves and have-nots of AI. "Intuitive physics" -- DeepMind claims progress towards AI with a better sense of context and "intuitive physics" via relational reasoning and visual prediction, but obstacles to human-like intelligence remain. Alternative schema -- While deep reinforcement learning (DRL) is all the rage right now, some organizations like Vicarious have taken alternative approaches such as their Schema Networks, which have outperformed some DRL nets albeit with some debate and controversy. Facebook fighting extremism with AI -- "The problem, as usual, is determining what is extremist, and what isn't, and it goes further than just jihadists," he said.

The success of deep reinforcement learning (deep RL) in playing games has resulted in a large amount of excitement in the AI community and beyond (Mnih et al., 2015; Mnih et al., 2016; Silver et al., 2016; Van Hasselt et al., 2016). State-of-the-art scores in many different games have now surpassed human level. But to what extent do these feats demonstrate that the AI has developed a human-like understanding of the objectives of the game? When humans play a new game, they first develop a conceptual understanding of the game. Suppose you were seeing a game like Breakout (see below) for the first time.

We propose an online, end-to-end, neural generative conversational model for open-domain dialogue. It is trained using a unique combination of offline two-phase supervised learning and online human-in-the-loop active learning. While most existing research proposes offline supervision or hand-crafted reward functions for online reinforcement, we devise a novel interactive learning mechanism based on hamming-diverse beam search for response generation and one-character user-feedback at each step. Experiments show that our model inherently promotes the generation of semantically relevant and interesting responses, and can be used to train agents with customized personas, moods and conversational styles.

At long last, the perfect score for arcade classic Ms. Pac-Man has been achieved, though not by a human. Maluuba -- a deep learning team acquired by Microsoft in January -- has created an AI system that's learned how to reach the game's maximum point value of 999,900 on Atari 2600, using a unique combination of reinforcement learning with a divide-and-conquer method. AI researchers have a documented penchant for using video games to test machine learning; they better mimic real-world chaos in a controlled environment versus more static games like chess. In 2015, Google's DeepMind AI was able to learn how to master 49 Atari games using reinforcement learning, which provides positive or negative feedback each time the AI attempts to solve a problem. Though AI has conquered a wealth of retro games, Ms. Pac-Man has remained elusive for years, due to the game's intentional lack of predictability.

Some tasks are just too complex, too nuanced to tackle all at once, like beating all 256 levels of Ms. Pac-Man on the Atari 2600 while earning a perfect score of 999,990. That's why Microsoft didn't even try to train its AI to take it on in one go. Instead the company, as it announced on Wednesday, split this monumental challenge up into smaller, chomp-sized pieces and trained a hivemind of 150 AIs to accomplish it as a team. Developed by Maluuba, a Canadian AI firm that Microsoft recently acquired, the AI system relies on reinforcement learning to develop its strategy. Reinforcement learning is an AI training technique wherein the algorithm is rewarded for using more efficient outcomes and dissuaded from using the less effective based on previously observed outcomes.

One step towards building safe AI systems is to remove the need for humans to write goal functions, since using a simple proxy for a complex goal, or getting the complex goal a bit wrong, can lead to undesirable and even dangerous behavior. In collaboration with DeepMind's safety team, we've developed an algorithm which can infer what humans want by being told which of two proposed behaviors is better. We present a learning algorithm that uses small amounts of human feedback to solve modern RL environments. Machine learning systems with human feedback have been explored before, but we've scaled up the approach to be able to work on much more complicated tasks. Our algorithm needed 900 bits of feedback from a human evaluator to learn to backflip -- a seemingly simple task which is simple to judge but challenging to specify.