The Long Journey Of Solving Gradient Problem

In the neural network training phase, especially with the backpropagation algorithm, we use the gradient descent principle to get the most minimal cost function possible. In linear regression, we can guarantee that our cost function will always converge because our cost function is convex. One and only problem in linear regression cases is if we set the learning rate too big. It will jump around and never touch slope equals to zero. If the learning rate is too small, it will eventually converge but it will take a lot of time.