Shuffling Recurrent Neural Networks

Recurrent Neural Networks (RNN) architectures have been successful in solving sequential or time dependent problems. Such methods maintain a latent representation, commonly referred to as the "hidden state", and apply the same learned functions repeatedly to the input at each time step, as well as to the current hidden state. A well-known challenge with RNNs, is that of exploding or vanishing gradients. The various methods that were devised in order to solve this problem can be roughly divided into two groups. The first group utilizes a gating mechanism to stabilize the gradient flow between subsequent hidden states [9, 4], whereas the second group focuses on preserving the norm of the hidden states by employing constraints on the family of matrices used as the network's parameters [1, 8]. An alternative view of the problem of exploding and vanishing gradient considers it as the symptom of a deeper issue and not as the root cause. Current RNN architectures perform a matrix multiplication operation, with learned weights, over previously seen hidden states during each time step. Therefore, inputs appearing in different times are processed using different powers of the weight matrices (with interleaving non-linearities): the first input of a sequence of length T is processed by the same learned sub-network T times, whereas the last input at time T is processed only once. This creates an inherent gap in the way that each time step influences the network weights during training.