Dual Averaging is Surprisingly Effective for Deep Learning Optimization

Stochastic first-order optimization methods have been extensively employed for training neural networks. It has been empirically observed that the choice of the optimization algorithm is crucial for obtaining a good accuracy score. For instance, stochastic variance-reduced methods perform poorly in computer vision (CV) (Defazio & Bottou, 2019). On the other hand, SGD with momentum (SGD M) (Bottou, 1991; LeCun et al., 1998; Bottou & Bousquet, 2008) works particularly well on CV tasks and Adam (Kingma & Ba, 2014) outperforms other methods on natural language processing (NLP) tasks (Choi et al., 2019). In general, the choice of optimizer, as well as its hyper-parameters, must be included among the set of hyper-parameters that are searched over when tuning.