A new trick for calculating Jacobian vector products

If you have any questions about this post please ask on the discussion thread on /r/machinelearning. For a solid introduction to Automatic Differentiation, which is the subject of this blog post, see Automatic differentiation in machine learning: a survey. Last week I was involved in a heated discussion thread over on the Autograd issue tracker. I'd recently been working on an implementation of forward mode automatic differentiation, which fits into Autograd's system for differentiating Python/Numpy code. Our discussion was about the usefulness of forward mode, which is equivalent to Theano's Rop and in the general case is used to calculate directional derivatives, or equivalently for calculating Jacobian vector products.