LSTMs

In past posts, I've described how Recurrent Neural Networks (RNNs) can be used to learn patterns in sequences of inputs, and how the idea of unrolling can be used to train them. It turns out that there are some significant limitations to the types of patterns that a typical RNN can learn, due to the way their weight matrices are used. As a result, there has been a lot of interest in a variant of RNNs called Long Short-Term Memory networks (LSTMs). As I'll describe below, LSTMs have more control than typical RNNs over what they remember, which allows them to learn much more complex patterns. Lets start with what I mean by a "typical" RNN.