Reviews: Predictive-State Decoders: Encoding the Future into Recurrent Networks

The authors present a simple method for regularizing recurrent neural networks (specifically applied to low-dimensional control tasks trained by filtering, imitation, and reinforcement learning) that is inspired by the literature on Predictive State Representations (PSRs). In PSRs, processes with latent states are modeled by directly identifying the latent state with a representation of the sufficient statistics of future observations. This stands in contrast to RNNs, which use an opaque hidden state that is optimized by backpropagation to make the whole system model the target of interest through evolution of the hidden state. They propose a simple regularization method for RNNs inspired by PSRs, called Predictive State Decoders, which maintains the parametric form of a standard RNN, but augments the training objective to make the hidden state more directly predict statistics of distant future observations. While direct training of an RNN by backpropagation implicitly encourages the internal state to be predictive of the future, the authors demonstrate that this explicit additional regularization term gives significant improvements on several control tasks.