Seven Myths in Machine Learning Research · Oscar Chang

It is actually a Matrix manipulation library, and this difference is significant. NeurIPS 2018., the authors demonstrate that their automatic differentiation library based on actual Tensor Calculus has significantly more compact expression trees. This is because Tensor Calculus uses index notation, which results in treating both the forward mode and the reverse mode in the same manner. By contrast, Matrix Calculus hides the indices for notational convenience, and this often results in overly complicated automatic differentiation expression trees. Consider the matrix multiplication .