Recurrent neural networks are powerful models for processing sequential data, but they are generally plagued by vanishing and exploding gradient problems. Unitary recurrent neural networks (uRNNs), which use unitary recurrence matrices, have recently been proposed as a means to avoid these issues. However, in previous experiments, the recurrence matrices were restricted to be a product of parameterized unitary matrices, and an open question remains: when does such a parameterization fail to represent all unitary matrices, and how does this restricted representational capacity limit what can be learned? To address this question, we propose full-capacity uRNNs that optimize their recurrence matrix over all unitary matrices, leading to significantly improved performance over uRNNs that use a restricted-capacity recurrence matrix. Our contribution consists of two main components. First, we provide a theoretical argument to determine if a unitary parameterization has restricted capacity. Using this argument, we show that a recently proposed unitary parameterization has restricted capacity for hidden state dimension greater than 7. Second,we show how a complete, full-capacity unitary recurrence matrix can be optimized over the differentiable manifold of unitary matrices. The resulting multiplicative gradient step is very simple and does not require gradient clipping or learning rate adaptation. We confirm the utility of our claims by empirically evaluating our new full-capacity uRNNs on both synthetic and natural data, achieving superior performance compared to both LSTMs and the original restricted-capacity uRNNs.

Neural Information Processing Systems http://nips.cc/

Unitary recurrent neural networks (URNNs) have been proposed as a method to overcome the vanishing and exploding gradient problem in modeling data with long-term dependencies. A basic question is how restrictive is the unitary constraint on the possible input-output mappings of such a network? This works shows that for any contractive RNN with ReLU activations, there is a URNN with at most twice the number of hidden states and the identical input-output mapping. Hence, with ReLU activations, URNNs are as expressive as general RNNs. In contrast, for certain smooth activations, it is shown that the input-output mapping of an RNN cannot be matched with a URNN, even with an arbitrary number of states. The theoretical results are supported by experiments on modeling of slowly-varying dynamical systems.

Recurrent neural networks are powerful models for processing sequential data, but they are generally plagued by vanishing and exploding gradient problems. Unitary recurrent neural networks (uRNNs), which use unitary recurrence matrices, have recently been proposed as a means to avoid these issues. However, in previous experiments, the recurrence matrices were restricted to be a product of parameterized unitary matrices, and an open question remains: when does such a parameterization fail to represent all unitary matrices, and how does this restricted representational capacity limit what can be learned? To address this question, we propose full-capacity uRNNs that optimize their recurrence matrix over all unitary matrices, leading to significantly improved performance over uRNNs that use a restricted-capacity recurrence matrix. Our contribution consists of two main components. First, we provide a theoretical argument to determine if a unitary parameterization has restricted capacity. Using this argument, we show that a recently proposed unitary parameterization has restricted capacity for hidden state dimension greater than 7. Second,we show how a complete, full-capacity unitary recurrence matrix can be optimized over the differentiable manifold of unitary matrices. The resulting multiplicative gradient step is very simple and does not require gradient clipping or learning rate adaptation. We confirm the utility of our claims by empirically evaluating our new full-capacity uRNNs on both synthetic and natural data, achieving superior performance compared to both LSTMs and the original restricted-capacity uRNNs.

Recurrent neural networks are powerful models for processing sequential data, but they are generally plagued by vanishing and exploding gradient problems. Unitary recurrent neural networks (uRNNs), which use unitary recurrence matrices, have recently been proposed as a means to avoid these issues.

When the transition matrix has an induced norm greater than one, the RNN may become unstable.

As shown in Figure 2 (top right), this solves the copying memory problem (Arjovsky et al., 2016).

UPDATE: I'm largely happy with how the authors addressed my points. I still think that the requirement for RNN to be non-expansive is quite restrictive per se, but this work may still be a good starting point for further theoretical discussion of such issues. The authors provide a straightforward proof by construction that a URNN with two times the number of hidden states as the corresponding RNN is as expressive as the RNN, i.e. can be formulated such that it produces the same outputs for the same series of inputs. While this is true for RNN with ReLU activation, the authors further prove, by linearizing around fixed points, that this is generally not true for RNN/URNN with sigmoid activation. Strengths: - Given that URNN are an important technique for modeling long-term dependencies, while avoiding some of the complexities of LSTM/GRU, rigorous theoretical results on how restrictive the unitary constraint is are timely and important.

I think this is a strong paper in that it presents multiple theoretical and empirical contributions. The theoretical ideas and proposed optimization algorithm are in my eyes more impressive than the empirical work, which is also decent but could benefit from a more thorough analysis. In any case, I think it's a very nice continuation of the ideas presented in the original paper about uRNNs. Except for some minor issues, the paper is well written in the sense that it was easy enough for me to follow the materials about Givens operators and how they can be used to quantify the representational capacity of a unitary matrix even though this specific subject matter was rather new to me. The proofs in the supplementary material seems sound to me and are relatively simple.