However, all known theoretical rates are worse than the rate of gradient descent, which synchronizes after every gradient step.

Recently, several instances of non-Euclidean SGD, including SignSGD, Lion, and Muon, have attracted significant interest from the optimization community due to their practical success in training deep neural networks. Consequently, a number of works have attempted to explain this success by developing theoretical convergence analyses. Unfortunately, these results cannot properly justify the superior performance of these methods, as they could not beat the convergence rate of vanilla Euclidean SGD. We resolve this important open problem by developing a new unified convergence analysis under the structured smoothness and gradient noise assumption. In particular, our results indicate that non-Euclidean SGD (i) can exploit the sparsity or low-rank structure of the upper bounds on the Hessian and gradient noise, (ii) can provably benefit from popular algorithmic tools such as extrapolation or momentum variance reduction, and (iii) can match the state-of-the-art convergence rates of adaptive and more complex optimization algorithms such as AdaGrad and Shampoo.

Andre Ward (30-0, 15 KOs) and Sergey Kovalev (30-0-1, 26 KOs) are among the best pound-for-pound boxers in the world, despite being nowhere close to being household names. The two light heavyweights will battle for Kovalev's WBA, IBF and WBO light heavyweight titles on Saturday night at T-Mobile Arena in Las Vegas in perhaps the most anticipated boxing event of 2016. Boxing purists are likely drooling over this fight because of the obvious similarities between the two superstars. They are the same height, have almost the same reach, have the same number of wins, fight from an orthodox stance, are fearless in the ring, and both fly under the radar despite their immense talent. Indeed, it's been a long time since there was a championship fight that pitted such evenly matched fighters.