MomentumRNN: Integrating Momentum into Recurrent Neural Networks

We also demonstrate that MomentumRNN is applicable to many types of recurrent cells, including those in the state-of-the-art orthogonal RNNs.