New State of the Art AI Optimizer: Rectified Adam (RAdam). Improve your AI accuracy instantly versus Adam, and why it works.

As you can see, RAdam provides a dynamic heuristic to provide automated variance reduction and thus removes the need and manual tuning involved with a warmup during training. In addition, RAdam is shown to be more robust to learning rate variations (the most important hyperparameter) and provides better training accuracy and generalization on a variety of datasets and within a variety of AI architectures. In short, I'd highly recommend you drop RAdam into your AI architecture and see if you don't get an immediate benefit. I'd offer a money back guarantee but since the cost for it is $0.00…:) RAdam is available for PyTorch at their official github here.