An overview of gradient descent optimisation algorithms

Gradient Descent is an optimisation algorithm used to find the optimum parameters (weights and biases) for a machine learning model such that it minimises a loss/cost function used to evaluate the performance of the model. Gradient descent is used when the parameters cannot be calculated analytically and thus have to be searched in the vast parameter space. Gradient descent is an iterative procedure that starts with a random set of parameters and continues to improve them slowly. To improve a given set of weights, we try to get the value of the cost function using the current weights (by calculating the gradient) and move in the direction in which the cost function reduces. Repeating this step for thousands of times, in most cases, gives us a set of weights the minimise the cost function.