[Deep Learning] Batch Normalization

Neural networks were difficult to train. Currently, we have a lot of tricks to achieve faster training and to solve the troubles that arise during model training. On the following, let's explore one of those tricks: batch normalization. When training a model, if the scale of two features (x1 1,2…, x2 100, 200 …) are quite different, the weight adjust in the small-scale features would be small. On the graph above, we observe that w1 just is adjusted slowly compared to w2 during the training.