The flurry of headlines surrounding AlphaGo Zero (the most recent version of DeepMind's AI system for playing Go) means interest in reinforcement learning (RL) is bound to increase. Next to deep learning, RL is among the most followed topics in AI. For most companies, RL is something to investigate and evaluate but few organizations have identified use cases where RL may play a role. As we enter 2018, I want to briefly describe areas where RL has been applied. RL is confusingly used to refer to a set of problems and a set of techniques, so let's first settle on what RL will mean for the rest of this post.

Check out the session "Get Your Hard Hat: Intelligent Industrial Systems with Deep Reinforcement Learning" at the AI Conference in Beijing, April 10-13, 2018. Best price ends January 26. The flurry of headlines surrounding AlphaGo Zero (the most recent version of DeepMind's AI system for playing Go) means interest in reinforcement learning (RL) is bound to increase. Next to deep learning, RL is among the most followed topics in AI. For most companies, RL is something to investigate and evaluate but few organizations have identified use cases where RL may play a role. As we enter 2018, I want to briefly describe areas where RL has been applied.

This is a long overdue blog post on Reinforcement Learning (RL). You may have noticed that computers can now automatically learn to play ATARI games (from raw game pixels!), they are beating world champions at Go, simulated quadrupeds are learning to run and leap, and robots are learning how to perform complex manipulation tasks that defy explicit programming. It turns out that all of these advances fall under the umbrella of RL research. I also became interested in RL myself over the last year: I worked through Richard Sutton's book, read through David Silver's course, watched John Schulmann's lectures, wrote an RL library in Javascript, over the summer interned at DeepMind working in the DeepRL group, and most recently pitched in a little with the design/development of OpenAI Gym, a new RL benchmarking toolkit. So I've certainly been on this funwagon for at least a year but until now I haven't gotten around to writing up a short post on why RL is a big deal, what it's about, how it all developed and where it might be going. It's interesting to reflect on the nature of recent progress in RL. Similar to what happened in Computer Vision, the progress in RL is not driven as much as you might reasonably assume by new amazing ideas. In Computer Vision, the 2012 AlexNet was mostly a scaled up (deeper and wider) version of 1990's ConvNets. Similarly, the ATARI Deep Q Learning paper from 2013 is an implementation of a standard algorithm (Q Learning with function approximation, which you can find in the standard RL book of Sutton 1998), where the function approximator happened to be a ConvNet. AlphaGo uses policy gradients with Monte Carlo Tree Search (MCTS) - these are also standard components.

Elon Musk has hired a new director of AI research at Tesla, and it may signal a plan to rethink the way its automated driving works. This week, Musk poached Andrej Karpathy, an expert on vision, deep learning, and reinforcement learning, from OpenAI, a nonprofit that Musk and others are funding that's dedicated to "discovering and enacting the path to safe artificial general intelligence." Karpathy, who will apparently report directly to Musk, is a rising star in the world of AI, having studied at Stanford with Fei-Fei Li, a leading AI expert who is now the chief scientist of Google Cloud. Li is famous in tech circles for having developed a data set of images that helped inspire a breakthrough in machine vision. Many have pointed to Karpathy's expertise in computer vision as a key asset for Tesla, and that's true.

A* finds the optimal path if the graph is known (you can describe exactly each state). Q-Learning (and Reinforcement Learning in general) tries to find the optimal path under unknown circumstances (part of the algorithm is to discover possible states, and often there are so many combinations that you can't learn all of them anyway) and in stochastic environments (action only leads to expected state with a certain probability). If you know the map of your country and want to navigate between cities, use A*. If you want to find the optimal order of steps (keystrokes) in a game, it's impossible to describe the problem as something A* could solve. Remember that A* uses heuristics to determine the next action.

Last week, Google's DeepMind team published a paper describing AlphaZero, a new generic reinforcement learning algorithm that has done some remarkable things. First, in about eight hours, it taught itself to beat AlphaGo, a human-trained AI system that beat the best human Go players in the world. It also taught itself chess and Shogi (known as Japanese chess) in about four hours and beat the best human-trained AI systems at those games. How did AlphaZero teach itself? The rules of the games were programmed into the system.

The next generation of AI applications will continuously interact with the environment and learn from these interactions. These applications impose new and demanding systems requirements, both in terms of performance and flexibility. In this paper, we consider these requirements and present Ray---a distributed system to address them. Ray implements a dynamic task graph computation model that supports both the task-parallel and the actor programming models. To meet the performance requirements of AI applications, we propose an architecture that logically centralizes the system's control state using a sharded storage system and a novel bottom-up distributed scheduler. In our experiments, we demonstrate sub-millisecond remote task latencies and linear throughput scaling beyond 1.8 million tasks per second. We empirically validate that Ray speeds up challenging benchmarks and serves as both a natural and performant fit for an emerging class of reinforcement learning applications and algorithms.

Reza Bosagh Zadeh is Founder CEO at Matroid and Adjunct Professor at Stanford University. His work focuses on Machine Learning, Distributed Computing, and Discrete Applied Mathematics. Reza received his PhD in Computational Mathematics from Stanford University under the supervision of Gunnar Carlsson. His awards include a KDD Best Paper Award and the Gene Golub Outstanding Thesis Award. He has served on the Technical Advisory Boards of Microsoft and Databricks.

Machine learning algorithms, and neural networks in particular, are considered to be the cause of a new AI'revolution'. In this article I will introduce the concept of reinforcement learning but with limited technical details so that readers with a variety of backgrounds can understand the essence of the technique, its capabilities and limitations. At the end of the article, I will provide links to a few resources for implementing RL. Broadly speaking, data-driven algorithms can be categorized into three types: Supervised, Unsupervised, and Reinforcement learning. The first two are generally used to perform tasks such as image classification, detection, etc.

Reinforcement learning (RL) is learning by interacting with an environment. An RL agent learns from the consequences of its actions, rather than from being explicitly taught and it selects its actions on basis of its past experiences (exploitation) and also by new choices (exploration), which is essentially trial and error learning. The reinforcement signal that the RL-agent receives is a numerical reward, which encodes the success of an action's outcome, and the agent seeks to learn to select actions that maximize the accumulated reward over time. In general we are following Marr's approach (Marr et al 1982, later re-introduced by Gurney et al 2004) by introducing different levels: the algorithmic, the mechanistic and the implementation level. The best studied case is when RL can be formulated as class of Markov Decision Problems (MDP). The agent can visit a finite number of states and in visiting a state, a numerical reward will be collected, where negative numbers may represent punishments.