r/MachineLearning - [D] Super-Convergence Skepticism

Smith and Topin's 2017 paper Super-Convergence: Very Fast Training of Neural Networks Using Large Learning Rates garnered quite a bit of attention, promising to cut training iterations by an order of magnitude without compromising accuracy. Using a 56-layer residual network, they claim that while it takes 80k iterations to train to 91% accuracy on CIFAR-10 using conventional algorithms, but that they can achieve a higher accuracy (92.4%) in only 10k iterations. On Open Review there is concern that it's not clear if the accuracy gains are significant ("no error bars") and about whether this technique generalizes to other architectures. I think we've mostly seen that Super-Convergence seems to converge to fine results on multiple architectures -- the "train ImageNet in 3 hours for $25" is probably the most well-known example. I've used it to train ResNet-18, 34, and 50 on CIFAR.