A major contributor to the quality of a deep learning model is the selection of the optimizer. We propose a new dual-joint search space in the realm of neural optimizer search (NOS), along with an integrity check, to automate the process of finding deep learning optimizers. Our dual-joint search space simultaneously allows for the optimization of not only the update equation, but also internal decay functions and learning rate schedules for optimizers. We search the space using our proposed mutation-only, particle-based genetic algorithm able to be massively parallelized for our domain-specific problem. We evaluate our candidate optimizers on the CIFAR-10 dataset using a small ConvNet. To assess generalization, the final optimizers were then transferred to large-scale image classification on CIFAR-100 and TinyImageNet, while also being fine-tuned on Flowers102, Cars196, and Caltech101 using EfficientNetV2Small. We found multiple optimizers, learning rate schedules, and Adam variants that outperformed Adam, as well as other standard deep learning optimizers, across the image classification tasks. Deep learning optimizers are built for solving optimization problems, where the goal is to find a set of parameters that optimizes a loss function in an efficient amount of time. The optimization landscapes of deep neural networks are vast and complex terrains with steep cliffs, saddle points, plateus, and valleys (Goodfellow et al., 2016). Being able to efficiently and intelligently maneuver across these landscapes is vital in order to achieve better performance and evaluation. The simplest way to update the weights of a network is through batch Stochastic Gradient Descent (SGD). With the goal of expediting convergence, adaptive methods have been created to efficiently scale the learning rate per parameter, such as RMSProp, AdaGrad (Duchi et al., 2011), and Adam (Kingma & Ba, 2017).

Deep Learning optimization involves minimizing a high-dimensional loss function in the weight space which is often perceived as difficult due to its inherent difficulties such as saddle points, local minima, ill-conditioning of the Hessian and limited compute resources. In this paper, we provide a comprehensive review of $14$ standard optimization methods successfully used in deep learning research and a theoretical assessment of the difficulties in numerical optimization from the optimization literature.

The choice of optimization algorithm can depend on several factors, including the specific problem you're trying to solve, the size and complexity of your dataset, and the resources (such as computational power and memory) that you have available.

Practical results have shown that deep learning optimizers using small constant learning rates, hyperparameters close to one, and large batch sizes can find the model parameters of deep neural networks that minimize the loss functions. We first show theoretical evidence that the momentum method (Momentum) and adaptive moment estimation (Adam) perform well in the sense that the upper bound of the theoretical performance measure is small with a small constant learning rate, hyperparameters close to one, and a large batch size. Next, we show that there exists a batch size called the critical batch size minimizing the stochastic first-order oracle (SFO) complexity, which is the stochastic gradient computation cost, and that SFO complexity increases once the batch size exceeds the critical batch size. Finally, we provide numerical results that support our theoretical results. That is, the numerical results indicate that Adam using a small constant learning rate, hyperparameters close to one, and the critical batch size minimizing SFO complexity has faster convergence than Momentum and stochastic gradient descent (SGD).

Is the Rectified Adam (RAdam) optimizer actually better than the standard Adam optimizer? According to my 24 experiments, the answer is no, typically not (but there are cases where you do want to use it instead of Adam). In Liu et al.'s 2018 paper, On the Variance of the Adaptive Learning Rate and Beyond, the authors claim that Rectified Adam can obtain: The authors tested their hypothesis on three different datasets, including one NLP dataset and two computer vision datasets (ImageNet and CIFAR-10). In each case Rectified Adam outperformed standard Adam…but failed to outperform standard Stochastic Gradient Descent (SGD)! The Rectified Adam optimizer has some strong theoretical justifications -- but as a deep learning practitioner, you need more than just theory -- you need to see empirical results applied to a variety of datasets. And perhaps more importantly, you need to obtain a mastery level experience operating/driving the optimizer (or a small subset of optimizers) as well. If you haven't yet, go ahead and read part one to ensure you have a good understanding of how the Rectified Adam optimizer works. From there, read today's post to help you understand how to design, code, and run experiments used to compare deep learning optimizers. To learn how to compare Rectified Adam to standard Adam, just keep reading! In the first part of this tutorial, we'll briefly discuss the Rectified Adam optimizer, including how it works and why it's interesting to us as deep learning practitioners.