A Glance at Optimization algorithms for Deep Learning

Batch Gradient Descent, Mini-batch Gradient Descent and Stochastic Gradient Descent are techniques used for gradient optimization differ in the batch size they use for computing gradients in each iteration. Gradient Descent uses all the data to compute gradients and update weights in each iteration. Minibatch Gradient Descent takes a subset of dataset to update its weights in each iteration. It however takes more iterations to converge to minima, but it is faster as compared to Gradient Descent due to lesser size of batch data used. Stochastic Gradient Descent (SGD) (or also sometimes on-line gradient descent) is the extreme case of this.