SVAIL Tech Notes: Optimizing RNNs with Differentiable Graphs - Baidu Research

This week we posted a new Tech Note in which Jesse Engel discusses a new technique for speeding up the training of deep recurrent neural networks. This is Part II of a multi-part series detailing some of the techniques we've used here at Baidu's Silicon Valley AI Lab (SVAIL) to accelerate the training of recurrent neural networks. While Part I focused on the role that minibatch and memory layout play on recurrent GEMM performance, we shift our focus here to tricks we can use to optimize the algorithms themselves. There are two main takeaways in this blog post. First, differentiable graphs are a simple and useful tool for visually calculating complicated derivatives.