Reinforcement Learning

In the previous couple of articles, we explored reinforcement learning ecosystem, how it can be described and how it functions. Reinforcement learning is a type of learning that is different from supervised and unsupervised learning. Unlike the mentioned approaches, reinforcement learning uses interaction, which makes it "the third paradigm of machine learning". Main reinforcement learning elements are learning agent and environment. These two elements are in constant interaction.

In the previous couple of articles, we explored reinforcement learning ecosystem, how it can be described and how it functions. Reinforcement learning is a type of learning that is different from supervised and unsupervised learning. Unlike the mentioned approaches, reinforcement learning uses interaction, which makes it "the third paradigm of machine learning". Main reinforcement learning elements are learning agent and environment. These two elements are in constant interaction.

DeepMind has been trying to bridge the gap between AI and biology for quite some time now. All their endeavours revolve around solving the problem of intelligence in machines. The straightforward trivial tasks for humans can be very, very sophisticated and almost for devices. While human brains are hardcoded with millions of years of learning, the machines have many limitations when it comes to data. They can be fed with data that has been documented or prepared by humans, the magnitude of which is historically insignificant when compared to humans.

A team of researchers from DeepMind, University College and Harvard University has found that lessons learned in applying learning techniques to AI systems may help explain how reward pathways work in the brain. In their paper published in the journal Nature, the group describes comparing distributional reinforcement learning in a computer with dopamine processing in the mouse brain, and what they learned from it. Prior research has shown that dopamine produced in the brain is involved in reward processing--it is produced when something good happens, and its expression results in feelings of pleasure. Some studies have also suggested that the neurons in the brain that respond to the presence of dopamine all respond in the same ways--an event causes a person or a mouse to feel either good or bad. Other studies have suggested that neuronal response is more of a gradient.

I wanted to name this Adventures in Reinforcement Learning. Then I realized that it was probably the lamest name I could ever create. Wikipedia can explain it better. Then why do you even have a blog post? Well I had to take a graduate level AI class to understand Reinforcement Learning enough for me to try playing around with examples I found online and tweak them to my interests, ultimately creating something in an hour while listening to RetroWave.

Ensemble learning is a very prevalent method employed in machine learning. The relative success of ensemble methods is attributed to its ability to tackle a wide range of instances and complex problems that require different low-level approaches. However, ensemble methods are relatively less popular in reinforcement learning owing to the high sample complexity and computational expense involved. We present a new training and evaluation framework for model-free algorithms that use ensembles of policies obtained from a single training instance. These policies are diverse in nature and are learned through directed perturbation of the model parameters at regular intervals.

It is difficult not to smile when reading the Wall Street Journal report about a guest in a robot-staffed hotel in Japan who was woken every few hours by the in-room assistant asking him to repeat his command. The hotel manager finally realized that heavy snoring by the guest had triggered the robot's voice recognition system. For every clanger, though, there is also a success story. For example, DeepMind's AI programme AlphaStar has for the first time beaten human video game players at StarCraft II, winning 10 games in a row. AlphaStar's success demonstrated the ability of AI programmes, in this case based on a reinforcement learning algorithm, to make quick decisions without any errors while operating in a complex environment.

Developments in artificial intelligence often draw inspiration from how humans think, but now AI has turned the tables to teach us about how brains learn. Will Dabney at tech firm DeepMind in London and his colleagues have found that a recent development in machine learning called distributional reinforcement learning also provides a new explanation for how the reward pathways in the brain work. These pathways govern our response to pleasurable events and are mediated by neurons that release the brain chemical dopamine. "Dopamine in the brain is a type of surprise signal," says Dabney. "When things turn out better than expected, more dopamine gets released."

Building deep reinforcement learning(DRL) systems remains an incredibly challenging. As a nascent discipline in the deep learning space, the frameworks and tools for implementing DRL models remain incredibly basic. Furthermore, the core innovation in DRL is coming from the big corporate AI labs like DeepMind, Facebook or Google. Almost a year ago, Facebook open sourced Horizon a framework focused on streamlining the implementation of DRL solutions. After a year using Horizon and implementing large scale DRL systems, Facebook open sourced ReAgent, a new framework that expands the original vision of Horizon to the implementation of end-to-end reasoning systems.

In recent years, the emergence of deep reinforcement learning (RL) has resulted in the growing demand for their evaluation. To implement and test RL models quickly and reliably, several RL libraries have been developed. Pyqlearning is a Python library to implement RL, especially for Q-Learning and multi-agent Deep Q-Network. This library makes it possible to design the information search algorithm such as the Game AI, web crawlers, or robotics. Keras-RL seamlessly implements state-of-the-art deep reinforcement learning algorithms with the deep learning library Keras.