Elon Musk's artificial intelligence company created virtual robots that can sumo wrestle and play soccer. Following is a transcript of the video. These AI robots are getting physical. They may look goofy but they're smarter than you think. OpenAI's bots can teach themselves how to sumo wrestle and play soccer.

Since its founding by Elon Musk and others nearly two years ago, nonprofit research lab OpenAI has published dozens of research papers. One posted online Thursday is different: Its lead author is still in high school. The wunderkind is Kevin Frans, a senior currently working on his college applications. He trained his first neural net--the kind of system that tech giants use to recognize your voice or face--two years ago, at the age of 15. Inspired by reports of software mastering Atari games and the board game Go, he has since been reading research papers and building pieces of what they described.

It happened with Chess and Go, and it finally happened with eSports. Elon Musk-backed Artificial Intelligence company "OpenAI" just used a bot to wallop the best DOTA2 players in the world. To be honest, it wasn't even close. Instead of trying to program the perfect bot, OpenAI just created a bot that learned through trial and error. Over the course of playing thousands of games against itself, the bot kept the behaviors that lead to victory and shed the ones that got it killed.

Starting with the Google DeepMind paper, there has been a lot of new attention around training models to play video games. You, the data scientist/engineer/enthusiast, may not work in reinforcement learning but probably are interested in teaching neural networks to play video games. The lessons below were gleaned from working on my own implementation of the Nature paper. The lessons are aimed at people who work with data but may run into some issues with some of the non-standard approaches used in the reinforcement learning community when compared with typical supervised learning use cases. I will address both technical details of the parameters of the neural networks and the libraries involved.

We propose a probabilistic model for interpreting gene expression levels that are observed through single-cell RNA sequencing. In the model, each cell has a low-dimensional latent representation. Additional latent variables account for technical effects that may erroneously set some observations of gene expression levels to zero. Conditional distributions are specified by neural networks, giving the proposed model enough flexibility to fit the data well. We use variational inference and stochastic optimization to approximate the posterior distribution. The inference procedure scales to over one million cells, whereas competing algorithms do not. Even for smaller datasets, for several tasks, the proposed procedure outperforms state-of-the-art methods like ZIFA and ZINB-WaVE. We also extend our framework to take into account batch effects and other confounding factors and propose a natural Bayesian hypothesis framework for differential expression that outperforms tradition DESeq2.

RoboSumo, one of the latest Open AI experiments in machine learning, involves a pair of'robots' dropped into a virtual arena without even the knowledge necessary to walk, and forced to learn the tricks of sumo wrestling purely by trial and error. The video posted on YouTube shows how the bots initially clash without employing any tactics or strategy, but after a number of bouts their movements start to resemble those of human wrestlers, as they learn to dodge and attack. According to the Wired, OpenAI researchers created RoboSumo because the competition apparently generated extra complexity which "could allow faster progress than just giving reinforcement learning software more complex problems to solve alone." "When you interact with other agents you have to adapt; if you don't you'll lose," Maruan Al-Shedivat, one of the RoboSumo creators, said.

We set up competitions between multiple simulated 3D robots on a range of basic games, trained each agent with simple goals (push the opponent out of the sumo ring, reach the other side of the ring while preventing the other agent from doing the same, kick the ball into the net or prevent the other agent from doing so, and so on), then analyzed the different strategies that emerged. Agents initially receive dense rewards for behaviours that aid exploration like standing and moving forward, which are eventually annealed to zero in favor of being rewarded for just winning and losing. Despite the simple rewards, the agents learn subtle behaviors like tackling, ducking, faking, kicking and catching, and diving for the ball. Each agent's neural network policy is independently trained with Proximal Policy Optimization. To understand how complex behaviors can emerge through a combination of simple goals and competitive pressure, let's analyze the sumo wrestling task.

Last time in our Keras/OpenAI tutorial, we discussed a very fundamental algorithm in reinforcement learning: the DQN. The Deep Q-Network is actually a fairly new advent that arrived on the seen only a couple years back, so it is quite incredible if you were able to understand and implement this algorithm having just gotten a start in the field. As with the original post, let's take a quick moment to appreciate how incredible results we achieved are: in a continuous output space scenario and starting with absolutely no knowledge on what "winning" entails, we were able to explore our environment and "complete" the trials. Put yourself in the situation of this simulation. This would essentially be like asking you to play a game, without a rulebook or specific endgoal, and demanding you to continue to play until you win (almost seems a bit cruel).

As part of its effort to find better ways to develop and train "safe artificial general intelligence," OpenAI has been releasing its own versions of reinforcement learning algorithms. They call these OpenAI Baselines, and the most recent additions to these algorithms are two baselines that are meant to enhance machine learning performance by making it more efficient. The first is a baseline implementation called Actor Critic using Kronecker-factored Trust Region (ACKTR). Developed by researchers from the University of Toronto (UofT) and New York University (NYU), ACKTR improves on the way AI policies perform deep reinforcement learning -- learning that is accomplished only by trial and error, and obtained only through raw observation. In a paper published online, the UofT and NYU researchers used simulated robots and Atari games to test how ACKTR learns control policies.

We've shown that deep linear networks -- as implemented using floating-point arithmetic -- are not actually linear and can perform nonlinear computation. We used evolution strategies to find parameters in linear networks that exploit this trait, letting us solve non-trivial problems. Neural networks consist of stacks of a linear layer followed by a nonlinearity like tanh or rectified linear unit. Without the nonlinearity, consecutive linear layers would be in theory mathematically equivalent to a single linear layer. So it's a surprise that floating point arithmetic is nonlinear enough to yield trainable deep networks. Numbers used by computers aren't perfect mathematical objects, but approximate representations using finite numbers of bits.