10 Gradient Descent Optimisation Algorithms – Towards Data Science

Gradient descent is an optimisation method for finding the minimum of a function. It is commonly used in deep learning models to update the weights of the neural network through backpropagation. In this post, I will summarise the common gradient descent optimisation algorithms that are used in popular deep learning frameworks (e.g. The purpose of this post is to make it easy to read and digest since there aren't many of such summaries out there, and as a cheat sheet if you want to implement them from scratch. I have implemented SGD, momentum, Nesterov, RMSprop and Adam in a linear regression problem using gradient descent demo here using JavaScript.