We present algorithms to effectively represent a set of Markov decision processes (MDPs), whose optimal policies have already been learned, by a smaller source subset for lifelong, policy-reuse-based transfer learning in reinforcement learning. This is necessary when the number of previous tasks is large and the cost of measuring similarity counteracts the benefit of transfer. The source subset forms an `$\epsilon$-net' over the original set of MDPs, in the sense that for each previous MDP $M_p$, there is a source $M^s$ whose optimal policy has $<\epsilon$ regret in $M_p$. Our contributions are as follows. We present EXP-3-Transfer, a principled policy-reuse algorithm that optimally reuses a given source policy set when learning for a new MDP. We present a framework to cluster the previous MDPs to extract a source subset. The framework consists of (i) a distance $d_V$ over MDPs to measure policy-based similarity between MDPs; (ii) a cost function $g(\cdot)$ that uses $d_V$ to measure how good a particular clustering is for generating useful source tasks for EXP-3-Transfer and (iii) a provably convergent algorithm, MHAV, for finding the optimal clustering. We validate our algorithms through experiments in a surveillance domain.

The big problem is that reinforcement learning is a difficult technique to characterise. Put simply an RL system learns not by being told how close it is the the desired result, but by receiving rewards based on its behaviour. Of course this is largely how we learn and if it can be made to work efficiently it promises us not just effective AI but new knowledge. For example AlphaGo taught itself to play Go and in the process discovered for itself approaches to Go that humans had ignored.

High-tech entrepreneur Elon Musk has launched an open-source training "gym" for artificial-intelligence programmers. It's an interesting move for a man who in 2014 said artificial intelligence, or A.I., will pose a threat to the human race. "I think we should be very careful about artificial intelligence," Musk said about a year and a half ago during an MIT symposium. "If I were to guess at what our biggest existential threat is, it's probably that... with artificial intelligence, we are summoning the demon. In all those stories with the guy with the pentagram and the holy water, and he's sure he can control the demon. Today, Musk is moving to help programmers use A.I. and machine learning to build smart robots and smart devices. "We're releasing the public beta of OpenAI Gym, a toolkit for developing and comparing reinforcement learning (RL) algorithms," wrote Greg Brockman, OpenAI's CTO, and John Schulman, a scientist working with OpenAI, in a blog post . "We originally built OpenAI Gym as a tool to accelerate our own RL research.

A few months ago, Elon Musk and a few other big players in Silicon Valley announced their creation of OpenAI, a non-profit research firm that would provide an open platform for researchers to share and compare algorithms for artificial intelligence. "Our goal is to advance digital intelligence in the way that is most likely to benefit humanity as a whole, unconstrained by a need to generate financial return," the team wrote in their introductory blog post. Now, the creators of OpenAI have announced a new platform for the organization, called OpenAI Gym. It's "a toolkit for developing and comparing reinforcement learning algorithms," as stated on the website. Interestingly, OpenAI Gym won't include leaderboards or competitions based on who can create the top scoring algorithm.

OpenAI, the nonprofit artificial intelligence research company established last year with backing from several Silicon Valley figures, today announced its first product: a proving ground for algorithms for reinforcement learning, which involves training machines to do things based on trial and error. OpenAI is releasing tools you can run locally to test out algorithms in various "environments" -- including Atari games like Air Raid, Breakout, and Ms. Pacman -- and a Web service for sharing test results. The system automatically scores evaluations and also seeks to have results reviewed and reproduced by other people. "We originally built OpenAI Gym as a tool to accelerate our own RL research. We hope it will be just as useful for the broader community," OpenAI's Greg Brockman and John Schulman wrote in a blog post. To be sure, there are other online places for showing off algorithms, including Algorithmia.

This tutorial doesn't seem to be linked from any of their pages, but was mentioned in their chat. It's very concise, but covers some key concepts related to Deep RL, and has exercises that can be implemented using the gym environment.

The design of multiple experiments is commonly undertaken via suboptimal strategies, such as batch (open-loop) design that omits feedback or greedy (myopic) design that does not account for future effects. This paper introduces new strategies for the optimal design of sequential experiments. First, we rigorously formulate the general sequential optimal experimental design (sOED) problem as a dynamic program. Batch and greedy designs are shown to result from special cases of this formulation. We then focus on sOED for parameter inference, adopting a Bayesian formulation with an information theoretic design objective. To make the problem tractable, we develop new numerical approaches for nonlinear design with continuous parameter, design, and observation spaces. We approximate the optimal policy by using backward induction with regression to construct and refine value function approximations in the dynamic program. The proposed algorithm iteratively generates trajectories via exploration and exploitation to improve approximation accuracy in frequently visited regions of the state space. Numerical results are verified against analytical solutions in a linear-Gaussian setting. Advantages over batch and greedy design are then demonstrated on a nonlinear source inversion problem where we seek an optimal policy for sequential sensing.

The next big trend in AI looks likely to be computers and robots that teach themselves through trial and error. Elon Musk and Sam Altman (of Y Combinator) caused a stir last December by luring several high-profile researchers to join OpenAI, a billion-dollar nonprofit dedicated to releasing cutting-edge artificial intelligence research for free. Today the nonprofit released the first fruits of its work, and it suggests that kind of learning will be important for the future of AI. The nonprofit has released a tool called OpenAI Gym for developing and comparing different so-called reinforcement learning algorithms, which provide a way for a machine to learn through positive and negative feedback. This week OpenAI also announced two new recruits, including Pieter Abbeel, an associate professor at Berkeley and a leading expert on applying reinforcement learning to robots. OpenAI Gym includes code and examples to help others get started with reinforcement learning.

NIPS 2015 was bigger than ever, literally: at circa 3700 attendees this was roughly twice as many attendees as last year, which in turn was roughly twice as many as the previous year. This is clearly unsustainable, but given the frenzied level of vendor and recruiting activities, perhaps there is room to grow. The main conference is single track, however, and already 3 days long: so even more action is moving to the poster sessions, which along with the workshops creates the feel of a diverse collection of smaller conferences. Obviously, my view of the action will be highly incomplete and biased towards my own interests. Reinforcement learning continues to ascend, extending the enthusiasm and energy from ICML.

This course will assume some familiarity with reinforcement learning, numerical optimization and machine learning. Students who are not familiar with the concepts below are encouraged to brush up using the references provided right below this list. We'll review this material in class, but it will be rather cursory. The assignments will be provided as Jupyter (formerly called IPython) notebooks (docs) and will use NumPy (docs) with Python 2.7. You may find the following tutorial helpful (from Stanford CS231): Python/Numpy.