Super-Convergence using Cyclical Learning Rate schedules is one of the most useful techniques in deep learning and very often overlooked. It allows for rapid prototyping of network architectures, loss function engineering, data augmentation experiments and training production ready models in orders of magnitude less training time and epochs. It is demonstrated within this article and the accompanying code that it is possible to train deep neural network models for complex tasks such as super resolution from initialised weights (i.e. The techniques for training in this way are from research that is a few years old. In 2015 Leslie Smith published a paper titled Cyclical Learning Rates for Training Neural Networks that outlines methods to train deep neural network models quicker (with fewer epochs) by cycling up to very high learning rates.

Deep neural networks are typically trained by optimizing a loss function with an SGD variant, in conjunction with a decaying learning rate, until convergence. We show that simple averaging of multiple points along the trajectory of SGD, with a cyclical or constant learning rate, leads to better generalization than conventional training. We also show that this Stochastic Weight Averaging (SWA) procedure finds much broader optima than SGD, and approximates the recent Fast Geometric Ensembling (FGE) approach with a single model. Using SWA we achieve notable improvement in test accuracy over conventional SGD training on a range of state-of-the-art residual networks, PyramidNets, DenseNets, and Shake-Shake networks on CIFAR-10, CIFAR-100, and ImageNet. In short, SWA is extremely easy to implement, improves generalization, and has almost no computational overhead.