AdaGrad does not adapt to Hölder-smoothness for composite objectives

Adaptive gradient methods are among the standard tools for training machine learning models. Their appeal is that they reduce the need to tune a fixed learning rate by adjusting the effective stepsize using information observed along the optimization trajectory. AdaGrad, introduced by Duchi et al. [2011], is a prototypical example: it rescales the update by the square root of the cumulative sum of past squared subgradients, coordinate by coordinate. The method was originally proposed for nonsmooth Lipschitz-continuous composite convex optimization, achieving the optimal rate O(1/ n) in the objective gap. Later works considered the smooth setting and asked whether AdaGrad can adapt to the unknown smoothness level of the objective, while attaining the corresponding standard rate.