The Unreasonable Effectiveness of Recurrent Neural Networks

Moreover, as we'll see in a bit, RNNs combine the input vector with their state vector with a fixed (but learned) function to produce a new state vector. If training vanilla neural nets is optimization over functions, training recurrent nets is optimization over programs. At the core, RNNs have a deceptively simple API: They accept an input vector x and give you an output vector y. Written as a class, the RNN's API consists of a single step function: The RNN class has some internal state that it gets to update every time step is called.