Reviews: Computing Higher Order Derivatives of Matrix and Tensor Expressions

This paper introduces a framework for efficient matrix and tensor differentiation. The main conceptual contribution, in comparison to existing automatic differentiation frameworks, is to work with expressions in Ricci calculus, which explicitly distinguish between covariant and contravariant indices. As tensor contraction is associative and commutative, this results in an elegant, expressive, and principled way to do automatic differentiation on tensor expressions, compatible with forward-mode, backward-mode, and symbolic differentiation. I believe this work is a useful and exciting contribution to the ML community at large. The authors have clearly put thoughtful and extensive engineering effort into this work, and go as far as to provide an anonymized web API for their implementation of symbolic differentiation using this framework.