Choose the Right Optimization Algorithm for your Neural Network

The development of any machine learning model is a highly iterative and empirical process that follows the idea-experiment-evaluation cycle. The cycle illustrated above is typically repeated multiple times before achieving satisfactory performances. The "experiment" phase includes the coding time and the training time of the machine learning model. As models' complexity increases and much more data are handled, training time inflates, with the result that training a very large deep neural network can be painfully slow [1]. While all the above-mentioned techniques are fundamental, this article details the last point.