Reviews: Reversible Recurrent Neural Networks

Summary: Authors investigate reversible architectures for RNN to reduce the memory requirement of their training. They build upon the reversible residual network architecture. By allowing reversibility, RNN can recompute the hidden activation during the backpropagation through time, trading computation for memory. Authors argue that a naïve reversible RNNs are not able to forget, hence have trouble to solve some simple learning tasks. To tackle this issue, they propose a variant, that store the lost information due to the forgetting.