Structured in Space, Randomized in Time: Leveraging Dropout in RNNs for Efficient Training

In this work, we identify dropout induced sparsity for LSTMs as a suitable mode of computation reduction. Dropout is a widely used regularization mechanism, which randomly drops computed neuron values during each iteration of training.