The Unreasonable Effectiveness of Recurrent Neural Networks

This articles was written by Andrej Karpathy. Andrej, PhD student at Stanford, is a Research Scientist at OpenAI working on Deep Learning, Generative Models and Reinforcement Learning. I still remember when I trained my first recurrent network for Image Captioning. Within a few dozen minutes of training my first baby model (with rather arbitrarily-chosen hyperparameters) started to generate very nice looking descriptions of images that were on the edge of making sense. Sometimes the ratio of how simple your model is to the quality of the results you get out of it blows past your expectations, and this was one of those times.