Gradient Descent
First Exit Time Analysis of Stochastic Gradient Descent Under Heavy-Tailed Gradient Noise
Thanh Huy Nguyen, Umut Simsekli, Mert Gurbuzbalaban, Gaël RICHARD
Neural Information Processing Systems http://nips.cc/
Exact natural gradient in deep linear networks and its application to the nonlinear case
Alberto Bernacchia, Mate Lengyel, Guillaume Hennequin
Neural Information Processing Systems http://nips.cc/
Natasha 2: Faster Non-Convex Optimization Than SGD
In diverse world of deep learning research has given rise to numerous architectures for neural networks(convolutionalones,longshorttermmemoryones,etc). However,tothisdate,theunderlying training algorithms for neural networks are still stochastic gradient descent (SGD) and its heuristic variants.
On the Global Convergence of (Fast) Incremental Expectation Maximization Methods
Belhal Karimi, Hoi-To Wai, Eric Moulines, Marc Lavielle
Neural Information Processing Systems http://nips.cc/
Uncertainty Sampling is Preconditioned Stochastic Gradient Descent on Zero-One Loss
Stephen Mussmann, Percy S. Liang
Neural Information Processing Systems http://nips.cc/
Parameter-free Clipped Gradient Descent Meets Polyak Y uki T akezawa
Gradient descent and its variants are de facto standard algorithms for training machine learning models. As gradient descent is sensitive to its hyperparame-ters, we need to tune the hyperparameters carefully using a grid search.
