Note on Learning Rate Schedules for Stochastic Optimization
Darken, Christian, Moody, John E.
–Neural Information Processing Systems
We present and compare learning rate schedules for stochastic gradient descent, a general algorithm which includes LMS, online backpropagation andk-means clustering as special cases. We introduce "search-thenconverge" typeschedules which outperform the classical constant and "running average" (1ft) schedules both in speed of convergence and quality of solution.
Neural Information Processing Systems
Dec-31-1991