Neural Network Optimization Algorithms – Towards Data Science

What are some of the popular optimization algorithms used for training neural networks? This article attempts to answer these questions using a Convolutional Neural Network (CNN) as an example trained on MNIST dataset with TensorFlow. The neural network is represented by f(x(i); theta) where x(i) are the training data and y(i) are the training labels, the gradient of the loss L is computed with respect to model parameters theta. The learning rate (eps_k) determines the size of the step that the algorithm takes along the gradient (in the negative direction in the case of minimization and in the positive direction in the case of maximization). The learning rate is a function of iteration k and is a single most important hyper-parameter.