Brain Damage On Artificial Intelligence

The vanishing gradient is one of the biggest challenges when training a deep neural network. It is a situation where a deep neural network is unable to backpropagate gradient from the output layer back to the first hidden layer. It often happens when we try to build a deep neural network with a sigmoid activation function on its hidden layers. The problem is that the sigmoid derivative is always less than 0. Given the formula above we can say that the biggest derivative is obtained when f(x) 0.5 so that f'(x) 0.5 * (1–0.5) which is 0.25. Now imagine when we try to build 7 layers neural network with a sigmoid activation function in each layer.