2 Deep Learning Methods Against Overfitting

Despite the fact that the batch normalization technique is aimed at preventing the problem of exploding gradients*, it also helps with the overfitting problem. This technique was described in detail in this work done by Ioffe, and Szeged about 6 years ago. Exploding gradients problem: If not controlled, the deep learning network weights can become too large, which would lead to the extreme values of the gradients, i.e., making gradients to explode making the loss function to return infinite/NaN values. Normalization is a process of scaling the data into a range of -1 to 1 or 0 to 1. Which is more like a matter of preference. This makes the input layer normalized, while the hidden layers are not normalized.