For iterative algorithms, implicit differentiation alleviates this issue but requires custom implementation of Jacobian evaluation. In this paper, we study one-step differentiation, also known as Jacobian-free backpropagation, a method as easy as automatic differentiation and as efficient as implicit differentiation for fast algorithms (e.g., superlinear

Stochastic gradient descent (SGD) algorithm is the method of choice in many machine learning tasks thanks to its scalability and efficiency in dealing with large-scale problems. In this paper, we focus on the shuffling version of SGD which matches the mainstream practical heuristics. We show the convergence to a global solution of shuffling SGD for a class of non-convex functions under over-parameterized settings.

This perspective enable a more direct comprehension of how the learning trajectory influences generalization error.

This perspective enable a more direct comprehension of how the learning trajectory influences generalization error.

Recently, there has been a marked increase in interest regarding the generalization capabilities of unregularized gradient-based learning methods.

Recently, there has been a marked increase in interest regarding the generalization capabilities of unregularized gradient-based learning methods.