Optimization algorithms in Deep Learning.

Deep Learning is one such field that is evolving continuously due to the availability of More Data, More computing, and more democratization. Recent developments in ChatGPT are one such example. Often while speaking about the training of Deep learning networks the obvious method that comes up is the gradient descent approach. As evolution continues to happen and due to the availability of more data, is the gradient descent the best approach or can there be any improvements or places to innovation for a new algorithm? For optimization of DL, there can few areas that can become a crucial point of discussion are better learning algorithms, better initialization techniques, better activations function, and better regularization techniques. But, in this article, we will touch base on some of the best optimization algorithms that are used for training/learning deep learning algorithms.