Thus, diagonal preconditioning methods remain popular.

We study the implicit bias of AdaGrad on separable linear classification problems. We show that AdaGrad converges to a direction that can be characterized as the solution of a quadratic optimization problem with the same feasible set as the hard SVM problem. We also give a discussion about how different choices of the hyperparameters of AdaGrad may impact this direction. This provides a deeper understanding of why adaptive methods do not seem to have the generalization ability as good as gradient descent does in practice.

Adaptive optimization methods, which perform local optimization with a metric constructed from the history of iterates, are becoming increasingly popular for training deep neural networks. Examples include AdaGrad, RMSProp, and Adam. We show that for simple overparameterized problems, adaptive methods often find drastically different solutions than gradient descent (GD) or stochastic gradient descent (SGD). We construct an illustrative binary classification problem where the data is linearly separable, GD and SGD achieve zero test error, and AdaGrad, Adam, and RMSProp attain test errors arbitrarily close to half. We additionally study the empirical generalization capability of adaptive methods on several state-of-the-art deep learning models. We observe that the solutions found by adaptive methods generalize worse (often significantly worse) than SGD, even when these solutions have better training performance. These results suggest that practitioners should reconsider the use of adaptive methods to train neural networks.

Over the past years data adaptiveness has proven to be crucial to the success of learning algorithms. The objective function underlying "big data" applications often demonstrates intricate structure: the scale and smoothness are often unknown and may change substantially in between different regions/directions, [

Examples include AdaGrad, RMSProp, and Adam.

In this work we propose such a method.

In this work we propose such a method.

Therefore, SGD requires hyperparameter tuning, which can be computationally expensive.

Our method retains the benefits of per-parameter adaptivity while allowing significantly larger models and batch sizes.