CS 230 - Recurrent Neural Networks Cheatsheet

Architecture of a traditional RNN ― Recurrent neural networks, also known as RNNs, are a class of neural networks that allow previous outputs to be used as inputs while having hidden states. Applications of RNNs ― RNN models are mostly used in the fields of natural language processing and speech recognition. Backpropagation through time ― Backpropagation is done at each point in time. Vanishing/exploding gradient ― The vanishing and exploding gradient phenomena are often encountered in the context of RNNs. The reason why they happen is that it is difficult to capture long term dependencies because of multiplicative gradient that can be exponentially decreasing/increasing with respect to the number of layers.