Goto

Collaborating Authors

 Deep Learning





Natasha 2: Faster Non-Convex Optimization Than SGD

Neural Information Processing Systems

In diverse world of deep learning research has given rise to numerous architectures for neural networks(convolutionalones,longshorttermmemoryones,etc). However,tothisdate,theunderlying training algorithms for neural networks are still stochastic gradient descent (SGD) and its heuristic variants.





a1e865a9b1065392ed6035d8ccd072d9-Paper.pdf

Neural Information Processing Systems

Unfortunately,the per-iteration cost of maintaining this adaptivedistribution for gradient estimation is more than calculating the full gradient itself, which we call the chicken-and-the-egg loop. As a result, the false impression of faster convergence in iterations, inreality,leads to slower convergence in time.