Reviews: Predictive State Recurrent Neural Networks

This paper proposes a new model for dynamical systems (called PSRNN), which combines the frameworks of PSR and RNN non-trivially. The model is learned from data in two steps: The first step initialize the model parameters using two-stage regression (2SR), a method previously proposed by Hefny et al for learning PSRs. The second step use Back-propagation-through-time to refine the parameters. The learned model can then be used for filtering and prediction. The model has an appealing bi-linear gating mechanism, resembling the non-linear gating mechanisms used in LSTM and other models and enjoys rich functional form via kernel embedding and/or multilayer stacking.