On the Performance of Preconditioned Stochastic Gradient Descent

Stochastic gradient descent (SGD) and its variations, e.g., SGD with either classic or Nesterov momentum, RMSProp, Adam, adaptive learning rates, etc., are popular in diverse stochastic optimization problems, e.g., machine learning and online signal processing [1]-[6]. Off-the-shelf methods from convex optimizations, e.g., the quasi-Newton method, conjugate gradient method and truncated Newton method, i.e., the Hessian-free optimization, are attracting more attentions [7]-[10], and find many successful applications in stochastic optimizations as well. At the same time, searchings for new optimization theories and learning rules are always active, and methods like natural gradient descent, relative gradient descent, and equilibrated SGD (ESGD) [11]-[13], provide us with great insight into the properties of the parameter spaces and cost function surfaces in stochastic optimizations. This paper studies the performance of preconditioned SGD (PSGD) [14], a method which explicitly considers the non-convexity and gradient noises in stochastic optimizations. We consider preconditioners in several forms, i.e., dense preconditioner, diagonal preconditioner, Kronecker product preconditioner and more flexible forms. ESGD and batch normalization [18] are shown to be PSGDs with specific forms of preconditioners. We also consider different ways to evaluate the Hessianvector product, an important measurement that helps PSGD to adaptively extract the curvature information of cost surfaces. Our benchmark problems have synthetic and real world data, and involves most commonly used neural network models, e.g., recurrent and convolutional networks. We argue that Kronecker product preconditioners are particularly suitable for training neural networks since affine transformations are their basic building blocks.