Abstract: Deep reinforcement learning algorithms have recently achieved impressive performance in playing video games. However, they require orders of magnitude more time than average human players to achieve the same performance. What makes humans so good at solving and figuring out these seemingly complex games? Here, we study one aspect critical to human decision making and problem solving – their use of strong priors (either learned or inbuilt), that helps them to generalize and solve tasks faster (as opposed to learning from scratch). Through systematic investigation of human performance in video games, we develop a taxonomy of different forms of prior knowledge employed by humans that enables them to quickly solve video games.

For the new year, we asked 18 of our researchers and staff members from around PARC to share some of their technology predictions for 2018 and beyond. "Deep learning and deep reinforcement learning has revolutionized data analytics, but the amount of data required is still relatively huge. It's also not how humans learn – typically a few examples are enough to form a mental model. I think this is going to change as research shifts from feasibility and accuracy to scalability and transparency. There's already some work in this area of slow learning, including from PARC, and this will only get more prominent."

Reinforcement learning (RL) has been successfully used to solve many continuous control tasks. Despite its impressive results however, fundamental questions regarding the sample complexity of RL on continuous problems remain open. We study the performance of RL in this setting by considering the behavior of the Least-Squares Temporal Difference (LSTD) estimator on the classic Linear Quadratic Regulator (LQR) problem from optimal control. We give the first finite-time analysis of the number of samples needed to estimate the value function for a fixed static state-feedback policy to within $\varepsilon$-relative error. In the process of deriving our result, we give a general characterization for when the minimum eigenvalue of the empirical covariance matrix formed along the sample path of a fast-mixing stochastic process concentrates above zero, extending a result by Koltchinskii and Mendelson in the independent covariates setting. Finally, we provide experimental evidence indicating that our analysis correctly captures the qualitative behavior of LSTD on several LQR instances.

In this chapter, we report the first experimental explorations of reinforcement learning in Tourette syndrome, realized by our team in the last few years. This report will be preceded by an introduction aimed to provide the reader with the state of the art of the knowledge concerning the neural bases of reinforcement learning at the moment of these studies and the scientific rationale beyond them. In short, reinforcement learning is learning by trial and error to maximize rewards and minimize punishments. A large body of evidence suggests that the dysfunction of the same neural systems is implicated in the pathophysiology of Tourette syndrome. Our results show that Tourette condition, as well as the most common pharmacological treatments (dopamine antagonists), affects reinforcement learning performance in these patients.

Sony has announced that it is strengthening its video and audio technologies through artificial intelligence (AI) and robotics by fusing its core offerings with the latest in technology. Together with its partner companies and research institutes, Sony is combining the traditional techniques of Japan with new technologies. The Sony Innovation Fund, a corporate venture capital which facilitates collaboration with leading external researches and start-up companies around the world, is focusing on AI and robotics to support the growth of investee businesses. This has resulted in investment in over 10 firms from Japan, the United States, Europe, and elsewhere. Sony's investment in Cogitai, an American start-up that specialises in AI, has also resulted in the development of AI technologies that use deep reinforcement learning and prediction technology as the basis for the next generation of AI applications and products.

Fuelled by successes in Computer Go, Monte Carlo tree search (MCTS) has achieved widespread adoption within the games community. Its links to traditional reinforcement learning (RL) methods have been outlined in the past; however, the use of RL techniques within tree search has not been thoroughly studied yet. In this paper we re-examine in depth this close relation between the two fields; our goal is to improve the cross-awareness between the two communities. We show that a straightforward adaptation of RL semantics within tree search can lead to a wealth of new algorithms, for which the traditional MCTS is only one of the variants. We confirm that planning methods inspired by RL in conjunction with online search demonstrate encouraging results on several classic board games and in arcade video game competitions, where our algorithm recently ranked first. Our study promotes a unified view of learning, planning, and search.

Using a new technique we invented to efficiently evolve DNNs, we were surprised to discover that an extremely simple genetic algorithm (GA) can train deep convolutional networks with over 4 million parameters to play Atari games from pixels, and on many games outperforms modern deep reinforcement learning (RL) algorithms (e.g. This result is surprising both because GAs, which are not gradient-based, were not expected to scale well to such large parameter spaces and also because matching or outperforming the state-of-the-art in RL using GAs was not thought to be possible. We further show that modern GA enhancements that improve the power of GAs, such as novelty search, also work at DNN scales and can promote exploration to solve deceptive problems (those with challenging local optima) that stymie reward-maximizing algorithms such as Q-learning (DQN), policy gradients (A3C), ES, and the GA.

Learning to Run–one of the five official challenges in the new NIPS Competition Track–focuses on Reinforcement learning environments with musculoskeletal models. Organized by Stanford Neuromuscular Biomechanics Laboratory, first prize is a NVIDIA DGX Station, the fastest personal supercomputer for researchers and data scientists.

Bayesian optimisation has been successfully applied to a variety of reinforcement learning problems. However, the traditional approach for learning optimal policies in simulators does not utilise the opportunity to improve learning by adjusting certain environment variables: state features that are unobservable and randomly determined by the environment in a physical setting but are controllable in a simulator. This paper considers the problem of finding a robust policy while taking into account the impact of environment variables. We present Alternating Optimisation and Quadrature (ALOQ), which uses Bayesian optimisation and Bayesian quadrature to address such settings. ALOQ is robust to the presence of significant rare events, which may not be observable under random sampling, but play a substantial role in determining the optimal policy. Experimental results across different domains show that ALOQ can learn more efficiently and robustly than existing methods.

While deep learning has had significant successes in computer vision thanks to the abundance of visual data, collecting sufficiently large real-world datasets for robot learning can be costly. To increase the practicality of these techniques on real robots, we propose a modular deep reinforcement learning method capable of transferring models trained in simulation to a real-world robotic task. We introduce a bottleneck between perception and control, enabling the networks to be trained independently, but then merged and fine-tuned in an end-to-end manner to further improve hand-eye coordination. On a canonical, planar visually-guided robot reaching task a fine-tuned accuracy of 1.6 pixels is achieved, a significant improvement over naive transfer (17.5 pixels), showing the potential for more complicated and broader applications. Our method provides a technique for more efficient learning and transfer of visuo-motor policies for real robotic systems without relying entirely on large real-world robot datasets.