Earlier this week, Google's DeepMind team published a paper describing AlphaZero, a new generic reinforcement learning algorithm that has done some remarkable things. First, in about eight hours, it taught itself to beat AlphaGo, a human-trained AI system that beat the best human Go players in the world. It also taught itself chess and Shogi (known as Japanese chess) in about four hours and beat the best human-trained AI systems at those games. How did AlphaZero teach itself? The rules of the games were programmed into the system.

Food group Danone is working with ToolsGroup on AI-enhanced software to improve the supply chain. Forecast error is down 20%. Product expiry is down 30 per cent. And there is a 55 per cent improvement in net uplift from promotions. These new beers are courtesy of London-based IntelligentX brewery which uses feedback to alter the recipe according to reinforcement learning and Bayesian optimisation system.

Deep reinforcement learning (DRL) has shown incredible performance in learning various tasks to the human level. However, unlike human perception, current DRL models connect the entire low-level sensory input to the state-action values rather than exploiting the relationship between and among entities that constitute the sensory input. Because of this difference, DRL needs vast amount of experience samples to learn. In this paper, we propose a Multi-focus Attention Network (MANet) which mimics human ability to spatially abstract the low-level sensory input into multiple entities and attend to them simultaneously. The proposed method first divides the low-level input into several segments which we refer to as partial states. After this segmentation, parallel attention layers attend to the partial states relevant to solving the task. Our model estimates state-action values using these attended partial states. In our experiments, MANet attains highest scores with significantly less experience samples. Additionally, the model shows higher performance compared to the Deep Q-network and the single attention model as benchmarks. Furthermore, we extend our model to attentive communication model for performing multi-agent cooperative tasks. In multi-agent cooperative task experiments, our model shows 20% faster learning than existing state-of-the-art model.

This paper aims at theoretically and empirically comparing two standard optimization criteria for Reinforcement Learning: i) maximization of the mean value and ii) minimization of the Bellman residual. For that purpose, we place ourselves in the framework of policy search algorithms, that are usually designed to maximize the mean value, and derive a method that minimizes the residual $\|T_* v_\pi - v_\pi\|_{1,\nu}$ over policies. A theoretical analysis shows how good this proxy is to policy optimization, and notably that it is better than its value-based counterpart. We also propose experiments on randomly generated generic Markov decision processes, specifically designed for studying the influence of the involved concentrability coefficient. They show that the Bellman residual is generally a bad proxy to policy optimization and that directly maximizing the mean value is much better, despite the current lack of deep theoretical analysis. This might seem obvious, as directly addressing the problem of interest is usually better, but given the prevalence of (projected) Bellman residual minimization in value-based reinforcement learning, we believe that this question is worth to be considered.

TD(0) is one of the most commonly used algorithms in reinforcement learning. Despite this, there is no existing finite sample analysis for TD(0) with function approximation, even for the linear case. Our work is the first to provide such results. Existing convergence rates for Temporal Difference (TD) methods apply only to somewhat modified versions, e.g., projected variants or ones where stepsizes depend on unknown problem parameters. Our analyses obviate these artificial alterations by exploiting strong properties of TD(0). We provide convergence rates both in expectation and with high-probability. The two are obtained via different approaches that use relatively unknown, recently developed stochastic approximation techniques.

The main aim of this paper is the development of easily verifiable sufficient conditions for stability (almost sure boundedness) and convergence of stochastic approximation algorithms (SAAs) with set-valued mean-fields, a class of model-free algorithms that have become important in recent times. In this paper we provide a complete analysis of such algorithms under three different, yet related sets of sufficient conditions, based on the existence of an associated global/local Lyapunov function. Unlike previous Lyapunov function based approaches, we provide a simple recipe for explicitly constructing the Lyapunov function, needed for analysis. Our work builds on the works of Abounadi, Bertsekas and Borkar (2002), Munos (2005), and Ramaswamy and Bhatnagar (2016). An important motivation for the flavor of our assumptions comes from the need to understand dynamic programming and reinforcement learning algorithms, that use deep neural networks (DNNs) for function approximations and parameterizations. These algorithms are popularly known as deep learning algorithms. As an important application of our theory, we provide a complete analysis of the stochastic approximation counterpart of approximate value iteration (AVI), an important dynamic programming method designed to tackle Bellman's curse of dimensionality. Further, the assumptions involved are significantly weaker, easily verifiable and truly model-free. The theory presented in this paper is also used to develop and analyze the first SAA for finding fixed points of contractive set-valued maps.

While reinforcement learning has achieved many successes, there are situations when it use is problematic. We describe the issues and how to work around them.

A GOOGLE supercomputer has created an "AI child" which can outperform its man-made rivals. The incredible machine named NASNet becomes smarter through "reinforcement learning" which sees it report back to its "parent" computer when completing tasks. The AI (artificial intelligence), which was created earlier this year, is able to recognise objects such as people and cars while watching real time video. NASNet is controlled by a neural network called AutoML which was created by humans at Google Brain. The parent AI teaches its offspring to do specific tasks which are repeated thousands of times.

One thing I was curious about is whether AlphaZero can play endgames. For example, a friend brought up whether AlphaZero could learn how to play Nim. For anybody who isn't familiar: https://en.wikipedia.org/wiki/Nim, the optimal strategy for Nim involves computing the xor of all the heap sizes. I thought no, largely due to the lack of gradient information/lack of structure/MCTS not being a good heuristic for the quality of the move. However, this game of Nim doesn't seem that different from say, a knight-bishop end game mating scenario for chess.