Super Convergence with Cyclical Learning Rates in TensorFlow

Super-Convergence using Cyclical Learning Rate schedules is one of the most useful techniques in deep learning and very often overlooked. It allows for rapid prototyping of network architectures, loss function engineering, data augmentation experiments and training production ready models in orders of magnitude less training time and epochs. It is demonstrated within this article and the accompanying code that it is possible to train deep neural network models for complex tasks such as super resolution from initialised weights (i.e. The techniques for training in this way are from research that is a few years old. In 2015 Leslie Smith published a paper titled Cyclical Learning Rates for Training Neural Networks that outlines methods to train deep neural network models quicker (with fewer epochs) by cycling up to very high learning rates.