For iterative algorithms, implicit differentiation alleviates this issue but requires custom implementation of Jacobian evaluation. In this paper, we study one-step differentiation, also known as Jacobian-free backpropagation, a method as easy as automatic differentiation and as efficient as implicit differentiation for fast algorithms (e.g., superlinear

It is desirable to restrict the number of components (i.e., encourage sparsity) for

However, FGD's dependencies across layers hinder parallel computation and can lead to inefficient resource utilization.

Both of the techniques assume bitwise access to the oracle as a classical function.

Computing gradients through backpropagation is crucial to the success of modern deep neural networks.

How biological neural networks learn in a supervised manner has long been an open problem.