Stochastic gradient descent (SGD) algorithm is the method of choice in many machine learning tasks thanks to its scalability and efficiency in dealing with large-scale problems. In this paper, we focus on the shuffling version of SGD which matches the mainstream practical heuristics. We show the convergence to a global solution of shuffling SGD for a class of non-convex functions under over-parameterized settings.

Besides, neural tangent kernel (NTK) based analysis is also given, which facilitates a comprehensive comparison. Our theory demonstrates the separation on the importance of different scaling schemes and initialization.

Neural Information Processing Systems http://nips.cc/

Neural Information Processing Systems http://nips.cc/

TD converges at a sublinear rate to the global optimum of the mean-squared projected Bellman error for policy evaluation.

The conditions on the representation that imply global convergence are different between these two algorithms.