Reinforcement Learning

Yet with attempt after attempt, Brett improves, learning by trial and error how to eventually nail the execution. A programmer could keep tweaking Brett's algorithm to get it to learn ever faster, sure. "And you might have a reinforcement learning algorithm that maybe can have a robot learn to walk in a few hours rather than two weeks, maybe even faster." Without robots learning to learn, humans will have to hold their hands.

Machine learning algorithms, and neural networks in particular, are considered to be the cause of a new AI'revolution'. In this algorithm, the agent learns the quality(Q value) of each action (action is also called policy) based on how much reward the environment gave it. As the agent interacts with the environment, the Q values get updated from random values to values that actually help maximize reward. When training a neural network, data imbalance plays a very important role.

So says Danny Lange, the VP of AI and machine learning at Unity Technologies, a major maker of game "engine" software that handles the underlying mechanics of titles like Firewatch and ChronoBlade. Today the company announced Unity Machine Learning Agents--open-source software linking its game engine to machine learning programs such as Google's TensorFlow. It will allow non-playable characters, through trial and error, to develop better, more creative strategies than a human could program, says Lange, using a branch of machine learning called deep reinforcement learning. And Nvidia's new Isaac Lab uses rival Epic Games' Unreal Engine to generate lifelike virtual environments for training the algorithms that control actual robots.

Today the company announced Unity Machine Learning Agents--open-source software linking its game engine to machine learning programs such as Google's TensorFlow. It will allow non-playable characters, through trial and error, to develop better, more creative strategies than a human could program, says Lange, using a branch of machine learning called deep reinforcement learning. Google's DeepMind, for instance, has used deep reinforcement learning to teach AI agents to play 1980s video games like Breakout, and, in part, to master the notoriously challenging ancient Chinese game Go. And Nvidia's new Isaac Lab uses rival Epic Games' Unreal Engine to generate lifelike virtual environments for training the algorithms that control actual robots.

The trick -- the company is calling it "concept networks" -- massively increases the efficiency of reinforcement learning. In a recently published paper, Bonsai's AI researchers describe how concept networks function by breaking out an objective into distinct problem areas. To teach a robot how to pick up and stack a block, for example, Bonsai has broken the task out into five concepts -- reach, orient, grasp, move and stack. DeepMind's paper describing its reinforcement learning approach takes on a similar grasping and stacking task with a robotic arm, but Bonsai's concept networks makes for a hugely more efficient system.

Several leading figures in AI, including LeCun, have studied or taught at Canadian universities. Reinforcement learning builds on deep learning to let machines learn through experimentation. Michael Bowling, a U.S.-born computer scientist who leads a lab at the University of Alberta that has produced cutting-edge poker-playing machines, says the new Facebook lab simply shows that Canada already leads the rest of the world in AI. Indeed, after seeing AI researchers snapped up by big U.S. companies in recent years, Canada may well hope that the environment fostered by new labs, including the one in Montreal, will eventually produce companies that rival the likes of Facebook.

"Gradient masking" is a term introduced in Practical Black-Box Attacks against Deep Learning Systems using Adversarial Examples. If the model's output is "99.9% airplane, 0.1% cat", then a little tiny change to the input gives a little tiny change to the output, and the gradient tells us which changes will increase the probability of the "cat" class. The defense strategies that perform gradient masking typically result in a model that is very smooth in specific directions and neighborhoods of training points, which makes it harder for the adversary to find gradients indicating good candidate directions to perturb the input in a damaging way for the model. Neither algorithm was explicitly designed to perform gradient masking, but gradient masking is apparently a defense that machine learning algorithms can invent relatively easily when they are trained to defend themselves and not given specific instructions about how to do so.

Deep Reinforcement Learning is a branch of Artificial Intelligence that allows machines to learn control and to take actions. The deterministic set of rules and abstract models used today to guide operations can account for such changes, but they cannot guarantee optimal performance. Reinforcement Learning agents learn from their experience of the environment, which helps them to see patterns we may miss and removes the need for modeling. Deep Reinforcement Learning can be deployed both centrally and on the edge at any level of the control chain: economic dispatch, battery dispatch, inverter control, cooling/boilers/combustors control, wind farm optimization and even outage detection.

Several leading figures in AI, including LeCun, have studied or taught at Canadian universities. Reinforcement learning builds on deep learning to let machines learn through experimentation. Michael Bowling, a U.S.-born computer scientist who leads a lab at the University of Alberta that has produced cutting-edge poker-playing machines, says the new Facebook lab simply shows that Canada already leads the rest of the world in AI. Indeed, after seeing AI researchers snapped up by big U.S. companies in recent years, Canada may well hope that the environment fostered by new labs, including the one in Montreal, will eventually produce companies that rival the likes of Facebook.

Testing our agents in games that are not specifically designed for AI research, and where humans play well, is crucial to benchmark agent performance. The bot learned the game from scratch by self-play, and does not use imitation learning or tree search. It beat players that many considered to be the absolute best at dota. However, there are cases where matchups do boil down to a 1v1 lane setup (at least for the first 10 minutes of the game), and the bot beat the players handily at it.