Summary: Are there large, sustainable career opportunities in AI and if so where? Do they lie in the current technologies of Deep Learning and Reinforcement Learning or should you focus your career on the next wave of AI? If you're a data scientist thinking about expanding your career options into AI you've got a forest and trees problem. There's a lot going on in deep learning and reinforcement learning but do these areas hold the best future job prospects or do we need to be looking a little further forward? To try to answer that question we'll have to get out of the weeds of current development and get a higher level perspective about where this is all headed. The roots of AI are actually in the behavioral sciences migrating eventually into biology and neurology.

Last year was huge for advancements in artificial intelligence and machine learning. But 2017 may well deliver even more. Here are five key things to look forward to. AlphaGo's historic victory against one of the best Go players of all time, Lee Sedol, was a landmark for the field of AI, and especially for the technique known as deep reinforcement learning. Reinforcement learning involves having a machine learn to solve a problem not through programming or explicit examples, but through experimentation combined with positive reinforcement.

We are currently seeking a hands-on Machine Learning Researcher (Reinforcement Learning) for our new research-led startup, focussing on the application of artificial intelligence in the real world; particularly smart city simulations and bots. We are looking for a Machine Learning Researcher who's fascinated with cutting-edge algorithms, who has a true understanding (theoretical & practical) of reinforcement learning or deep learning. As a Machine Learning Researcher, you will be working alongside the engineering learning team; providing them with research that will change how we interact with bots in the future. You will build, alter and evolve machine learning algorithms as you become part of our vision to help society through our work. Ideally as a Machine Learning Researcher you will come from a research background; be that a PhD in Data Science or Machine Learning or another research capacity.

Definitely! Asynchronous Deep Reinforcement Learning has really set a new standard to Deep Reinforcement Learning. One of its most important features is that you train a "global" model asynchronously through several agents, which basically means that you can have anywhere in between 8-32 agents training and sharing a global model at the same time. It requires far less memory than e.g. Experience Replay and also allows the agent to potentially converge much faster as it explores a broader state space asynchronously.

Scalable and effective exploration remains a key challenge in reinforcement learning (RL). While there are methods with optimality guarantees in the setting of discrete state and action spaces, these methods cannot be applied in high-dimensional deep RL scenarios. As such, most contemporary RL relies on simple heuristics such as epsilon-greedy exploration or adding Gaussian noise to the controls. This paper introduces Variational Information Maximizing Exploration (VIME), an exploration strategy based on maximization of information gain about the agent's belief of environment dynamics. We propose a practical implementation, using variational inference in Bayesian neural networks which efficiently handles continuous state and action spaces. VIME modifies the MDP reward function, and can be applied with several different underlying RL algorithms. We demonstrate that VIME achieves significantly better performance compared to heuristic exploration methods across a variety of continuous control tasks and algorithms, including tasks with very sparse rewards.

Guided policy search algorithms can be used to optimize complex nonlinear policies, such as deep neural networks, without directly computing policy gradients in the high-dimensional parameter space. Instead, these methods use supervised learning to train the policy to mimic a "teacher" algorithm, such as a trajectory optimizer or a trajectory-centric reinforcement learning method. Guided policy search methods provide asymptotic local convergence guarantees by construction, but it is not clear how much the policy improves within a small, finite number of iterations. We show that guided policy search algorithms can be interpreted as an approximate variant of mirror descent, where the projection onto the constraint manifold is not exact. We derive a new guided policy search algorithm that is simpler and provides appealing improvement and convergence guarantees in simplified convex and linear settings, and show that in the more general nonlinear setting, the error in the projection step can be bounded. We provide empirical results on several simulated robotic navigation and manipulation tasks that show that our method is stable and achieves similar or better performance when compared to prior guided policy search methods, with a simpler formulation and fewer hyperparameters.

Consider learning a policy from example expert behavior, without interaction with the expert or access to a reinforcement signal. One approach is to recover the expert's cost function with inverse reinforcement learning, then extract a policy from that cost function with reinforcement learning. This approach is indirect and can be slow. We propose a new general framework for directly extracting a policy from data as if it were obtained by reinforcement learning following inverse reinforcement learning. We show that a certain instantiation of our framework draws an analogy between imitation learning and generative adversarial networks, from which we derive a model-free imitation learning algorithm that obtains significant performance gains over existing model-free methods in imitating complex behaviors in large, high-dimensional environments.

Most learning algorithms are not invariant to the scale of the signal that is being approximated. We propose to adaptively normalize the targets used in the learning updates. This is important in value-based reinforcement learning, where the magnitude of appropriate value approximations can change over time when we update the policy of behavior. Our main motivation is prior work on learning to play Atari games, where the rewards were clipped to a predetermined range. This clipping facilitates learning across many different games with a single learning algorithm, but a clipped reward function can result in qualitatively different behavior. Using adaptive normalization we can remove this domain-specific heuristic without diminishing overall performance.

Feature construction is of vital importance in reinforcement learning, as the quality of a value function or policy is largely determined by the corresponding features. The recent successes of deep reinforcement learning (RL) only increase the importance of understanding feature construction. Typical deep RL approaches use a linear output layer, which means that deep RL can be interpreted as a feature construction/encoding network followed by linear value function approximation. This paper develops and evaluates a theory of linear feature encoding. We extend theoretical results on feature quality for linear value function approximation from the uncontrolled case to the controlled case. We then develop a supervised linear feature encoding method that is motivated by insights from linear value function approximation theory, as well as empirical successes from deep RL. The resulting encoder is a surprisingly effective method for linear value function approximation using raw images as inputs.

Efficient exploration remains a major challenge for reinforcement learning (RL). Common dithering strategies for exploration, such as epsilon-greedy, do not carry out temporally-extended (or deep) exploration; this can lead to exponentially larger data requirements. However, most algorithms for statistically efficient RL are not computationally tractable in complex environments. Randomized value functions offer a promising approach to efficient exploration with generalization, but existing algorithms are not compatible with nonlinearly parameterized value functions. As a first step towards addressing such contexts we develop bootstrapped DQN. We demonstrate that bootstrapped DQN can combine deep exploration with deep neural networks for exponentially faster learning than any dithering strategy. In the Arcade Learning Environment bootstrapped DQN substantially improves learning speed and cumulative performance across most games.