Backpropogating an LSTM: A Numerical Example

We all know LSTM's are super powerful; So, we should know how they work and how to use them. Above is the element-wise product or Hadamard product. I'm using a sequence length of two here to demonstrate the unrolling over time of RNNs From here, we can pass forward our state and output and begin the next time-step. And since we're done our sequence we have everything we need to begin backpropogating. First we'll need to compute the difference in output from the expected (label).