2017: The Year of Neuroevolution

This month OpenAI published a paper "Evolution Strategies as a Scalable Alternative to Reinforcement Learning" by Tim Salimans, Jonathan Ho, Xi Chen, Ilya Sutskever which shows Evolution Strategies (ES) can be a strong alternative to Reinforcement Learning (RL) and have a number of advantages like ease of implementation, invariance to the length of the episode and settings with sparse rewards, better exploration behaviour than policy gradient methods, ease to scale in a distributed setting. Running on a computing cluster of 80 machines and 1,440 CPU cores, authors' implementation was able to train a 3D MuJoCo humanoid walker in only 10 minutes (A3C on 32 cores takes about 10 hours). Using 720 cores they can also obtain comparable performance to A3C on Atari while cutting down the training time from 1 day to 1 hour. The communication overhead of implementing ES in a distributed setting is lower than for reinforcement learning methods such as policy gradients and Q-learning. By not requiring backpropagation, black box optimizers (the ones make no assumptions about the structure of the function being optimized) reduce the amount of computation per episode by about two thirds, and memory by potentially much more.