Neural Optimizer Equation, Decay Function, and Learning Rate Schedule Joint Evolution

A major contributor to the quality of a deep learning model is the selection of the optimizer. We propose a new dual-joint search space in the realm of neural optimizer search (NOS), along with an integrity check, to automate the process of finding deep learning optimizers. Our dual-joint search space simultaneously allows for the optimization of not only the update equation, but also internal decay functions and learning rate schedules for optimizers. We search the space using our proposed mutation-only, particle-based genetic algorithm able to be massively parallelized for our domain-specific problem. We evaluate our candidate optimizers on the CIFAR-10 dataset using a small ConvNet. To assess generalization, the final optimizers were then transferred to large-scale image classification on CIFAR-100 and TinyImageNet, while also being fine-tuned on Flowers102, Cars196, and Caltech101 using EfficientNetV2Small. We found multiple optimizers, learning rate schedules, and Adam variants that outperformed Adam, as well as other standard deep learning optimizers, across the image classification tasks. Deep learning optimizers are built for solving optimization problems, where the goal is to find a set of parameters that optimizes a loss function in an efficient amount of time. The optimization landscapes of deep neural networks are vast and complex terrains with steep cliffs, saddle points, plateus, and valleys (Goodfellow et al., 2016). Being able to efficiently and intelligently maneuver across these landscapes is vital in order to achieve better performance and evaluation. The simplest way to update the weights of a network is through batch Stochastic Gradient Descent (SGD). With the goal of expediting convergence, adaptive methods have been created to efficiently scale the learning rate per parameter, such as RMSProp, AdaGrad (Duchi et al., 2011), and Adam (Kingma & Ba, 2017).