Is Rectified Adam actually *better* than Adam? - PyImageSearch

Is the Rectified Adam (RAdam) optimizer actually better than the standard Adam optimizer? According to my 24 experiments, the answer is no, typically not (but there are cases where you do want to use it instead of Adam). In Liu et al.'s 2018 paper, On the Variance of the Adaptive Learning Rate and Beyond, the authors claim that Rectified Adam can obtain: The authors tested their hypothesis on three different datasets, including one NLP dataset and two computer vision datasets (ImageNet and CIFAR-10). In each case Rectified Adam outperformed standard Adam…but failed to outperform standard Stochastic Gradient Descent (SGD)! The Rectified Adam optimizer has some strong theoretical justifications -- but as a deep learning practitioner, you need more than just theory -- you need to see empirical results applied to a variety of datasets. And perhaps more importantly, you need to obtain a mastery level experience operating/driving the optimizer (or a small subset of optimizers) as well. If you haven't yet, go ahead and read part one to ensure you have a good understanding of how the Rectified Adam optimizer works. From there, read today's post to help you understand how to design, code, and run experiments used to compare deep learning optimizers. To learn how to compare Rectified Adam to standard Adam, just keep reading! In the first part of this tutorial, we'll briefly discuss the Rectified Adam optimizer, including how it works and why it's interesting to us as deep learning practitioners.