The Vanishing Gradient Problem – Anish Singh Walia – Medium

Vanishing Gradient Problem occurs when we try to train a Neural Network model using Gradient based optimization techniques. Vanishing Gradient Problem was actually a major problem 10 years back to train a Deep neural Network Model due to the long training process and the degraded accuracy of the Model. What happens is that as we keep on adding more and more Hidden layers in The model, the learning speed of the next hidden layers in the model keep on getting faster and faster. Generally, adding more hidden layers tends to make the network able to learn more complex arbitrary functions, and thus do a better job in predicting future outcomes. This is where Deep Learning is making a big difference due to the thousands and millions of hidden layers it has, we can now make sense of highly complicated data such as images, speeches, videos etc and do Speech Recognition and Image Classification, Image Captioning etc. Now when we do Back-propagation i.e moving backward in the Network and calculating gradients of loss(Error) with respect to the weights, the gradients tends to get smaller and smaller as we keep on moving backward in the Network.