Optimization is an integral part of most machine learning systems and most numerical optimization schemes rely on the computation of derivatives. Therefore, frameworks for computing derivatives are an active area of machine learning research. Surprisingly, as of yet, no existing framework is capable of computing higher order matrix and tensor derivatives directly. Here, we close this fundamental gap and present an algorithmic framework for computing matrix and tensor derivatives that extends seamlessly to higher order derivatives. The framework can be used for symbolic as well as for forward and reverse mode automatic differentiation. Experiments show a speedup between one and four orders of magnitude over state-of-the-art frameworks when evaluating higher order derivatives.

This paper introduces a framework for efficient matrix and tensor differentiation. The main conceptual contribution, in comparison to existing automatic differentiation frameworks, is to work with expressions in Ricci calculus, which explicitly distinguish between covariant and contravariant indices. As tensor contraction is associative and commutative, this results in an elegant, expressive, and principled way to do automatic differentiation on tensor expressions, compatible with forward-mode, backward-mode, and symbolic differentiation. I believe this work is a useful and exciting contribution to the ML community at large. The authors have clearly put thoughtful and extensive engineering effort into this work, and go as far as to provide an anonymized web API for their implementation of symbolic differentiation using this framework.

Optimization is an integral part of most machine learning systems and most numerical optimization schemes rely on the computation of derivatives. Therefore, frameworks for computing derivatives are an active area of machine learning research. Surprisingly, as of yet, no existing framework is capable of computing higher order matrix and tensor derivatives directly. Here, we close this fundamental gap and present an algorithmic framework for computing matrix and tensor derivatives that extends seamlessly to higher order derivatives. The framework can be used for symbolic as well as for forward and reverse mode automatic differentiation.