When Recurrent Models Don't Need to be Recurrent

An earlier version of this post was published on Off the Convex Path. It is reposted here with the author's permission. In the last few years, deep learning practitioners have proposed a litany of different sequence models. Although recurrent neural networks were once the tool of choice, now models like the autoregressive Wavenet or the Transformer are replacing RNNs on a diverse set of tasks. In this post, we explore the trade-offs between recurrent and feed-forward models.