Reviews: Approximating Real-Time Recurrent Learning with Random Kronecker Factors

Summary: This paper investigates Real-Time Recurrent Learning (RTRL) for training recurrent neural networks. As RTRL has a high-computational/memory costs, authors propose to approximate RTRL using the Kronecker Factorization. For a subclass of RNN, authors demonstrate that their approximation, KF-RTRL, is an unbiased estimate of the true gradient and has lower variance than previously proposed RTRL approximation such as UORO (relying on a low-rank approximation). KF-RTRL is empirically evaluated on a copy and language modeling task. Authors validate that KF-RTRL is a competitive alternative to UORO, also it underperforms truncated Backpropagation through time.