Optimizing Deep Neural Networks through Hyperparameter Tuning.

Learning rate is a hyperparameter with a small positive value (often range between 0.0 to 1.0) which controls how quickly the model adapts to the problem, in other words learning rates decides how far the weights should be in the direction of gradient to meet the global minimum. If the learning rate is too low (figure 1) training will progress very slowly. If the learning rate is set too high as it is shown in figure 3, training may not converge at the global minimum instead overshoots it and continue to make the loss of the model even worse. Getting the right learning rate at once is not possible (unless you are extremely experienced and lucky at the same time). As it is with all the hyperparameters it is a trial and error scenario.