The post coincides topically with last years' first annual Conference on Robot Learning as well as the workshop on Challenges in Robot Learning at NIPS2017, the latter we had the pleasure of co-organising together with colleagues from Oxford, DeepMind, and MIT. The events, as well as this post, cover current challenges and potentials of learning across various tasks of relevance in robotics and automation. In this context, similar to the long-term discussion on how much innate structure is optimal for artificial general intelligence, there is the more short-term question of how to merge traditional programming and learning (not sure if I prefer the branding as differentiable programming or software 2.0) for more narrow applications in efficient, robust and safe automation. The question about structure as beneficial or limiting aspect becomes arguably easier to answer in the context of robotic near-term applications as we can simply acknowledge our ignorance (our missing knowledge about what will work best in the future) and focus on the present to benchmark and combine the most efficient and effective directions. Existing solutions to many tasks in mobile robotics, such as localisation, mapping, or planning, focus on prior knowledge about the structure of our tasks and environments. This may include geometry or kinematic and dynamic models, which therefore have been built into traditional programs. However, recent successes and the flexibility of fairly unconstrained, learned models shift the focus of new academic and industrial projects. Successes in image recognition (ImageNet) as well as triumphs in reinforcement learning (Atari, Go, Chess) inspire like-minded research. As the post has become a bit of a long read, I suggest to read it like a paper: intro, discussion & conclusions and then - only if you did not fall asleep after all - the rest. Similar to scientific papers, some paragraphs will require basic familiarity with the field. However, a coarse web search should be enough to illustrate most unexplained terminology. Additionally, to keep this engaging, I have added some of my favourite recent videos highlighting interesting research for each section. Finally, this is a high-level review with more details to be found in the respective references, which just represent a small subset of available work in each field, chosen based on personal interest as well as shameless self-promotion of our work.

Recent news from the OpenAI people is all about a bonus trio. They are releasing new Gym environments--a set of simulated robotics environments based on real robot platforms--including a Shadow hand and a Fetch research robot, said IEEE Spectrum. In addition to that toolkit, they are releasing an open source version of Hindsight Experience Replay (HER). As its name suggests, it helps robots learn from hindsight, for goals-based robotic tasks. Last but not least, they released a set of requests for robotics research.

Today we're going to be learning about reinforcement learning. The ultimate goal of this endeavor is to create an artificial intelligence that is a strong Othello player, and can teach you how to become stronger yourself. I explained the rules of Othello, my motivation, and how to create a playable game in Step 1 of this series. I created some basic artificial intelligence in Step 2 of this series. The next thing I want to do is to use machine learning to create an even better artificial intelligence, but before I can even do that, I need to learn how to implement reinforcement learning. In many machine learning aspects, the cause/effect relationship is fairly obvious: A house with X square feet, Y bedrooms, and Z bathrooms in neighborhood Q will cost, on average N dollars.

Can we land a SpaceX Falcon Heavy Rocket in simulation using machine learning? Yes! Reinforcement learning is a technique that lets an agent learn how best to act in an environment using rewards as its signal. OpenAI released a library called Gym that lets us train AI agents really easily. We'll use a combination of the Tensorflow and gym libraries to build an RL agent capable of landing a rocket perfectly. The specific technique we're using is called proximal policy optimization, this is an actor-critic algorithm that is really popular.

Doubly stochastic learning algorithms are scalable kernel methods that perform very well in practice. However, their generalization properties are not well understood and their analysis is challenging since the corresponding learning sequence may not be in the hypothesis space induced by the kernel. In this paper, we provide an in-depth theoretical analysis for different variants of doubly stochastic learning algorithms within the setting of nonparametric regression in a reproducing kernel Hilbert space and considering the square loss. Particularly, we derive convergence results on the generalization error for the studied algorithms either with or without an explicit penalty term. To the best of our knowledge, the derived results for the unregularized variants are the first of this kind, while the results for the regularized variants improve those in the literature. The novelties in our proof are a sample error bound that requires controlling the trace norm of a cumulative operator, and a refined analysis of bounding initial error.

In recent months, researchers at OpenAI have been focusing on developing artificial intelligence (AI) that learns better. Their machine learning algorithms are now capable of training themselves, so to speak, thanks to the reinforcement learning methods of their OpenAI Baselines. Now, a new algorithm lets their AI learn from its own mistakes, almost as human beings do. The development comes from a new open-source algorithm called Hindsight Experience Replay (HER), which OpenAI researchers released earlier this week. As its name suggests, HER helps an AI agent "look back" in hindsight, so to speak, as it completes a task.

Which authors of this paper are endorsers? Disable MathJax (What is MathJax?)

Disclaimer: This is a work in progress project there may be errors! In order to fast recap my knowledge of Reinforcement Learning, I created this Cheat Sheet with all the basic formulas and algorithms. I hope this may be useful to you. You can find the full pdf here, and the repo here.

Modern learning systems, such as the recent deep learning, reinforcement learning, and probabilistic inference architectures, have become increasingly complex, often beyond the human ability to comprehend them. Such complexity is important: The more complex these systems are, the more powerful they often are. A new research problem has therefore emerged: How can the complexity, i.e. the design, components, and hyperparameters, be configured automatically so that these systems perform as well as possible? This is the problem of metalearning. Several approaches have emerged, including those based on Bayesian optimization, gradient descent, reinforcement learning, and evolutionary computation.

This paper presents a new multi-objective deep reinforcement learning (MODRL) framework based on deep Q-networks. We propose linear and non-linear methods to develop the MODRL framework that includes both single-policy and multi-policy strategies. The experimental results on a deep sea treasure environment indicate that the proposed approach is able to converge to the optimal Pareto solutions. The proposed framework is generic, which allows implementation of different deep reinforcement learning algorithms in various complex environments. Details of the framework implementation can be referred to http://www.deakin.edu.au/~thanhthi/drl.htm.