Why Is It So Hard To Train Neural Networks?

Neural networks are hard to train. The more they go deep, the more they are likely to suffer from unstable gradients. Gradients can either explode or vanish, and neither of those is a good thing for the training of our network. The vanishing gradients problem results in the network taking too long to train(learning will be very slow or completely die), and the exploding gradients cause the gradients to be very large. Although those problems are nearly inevitable, the choice of activation function can reduce their effects. Using ReLU activation in the first layers can help avoid vanishing gradients.