[DNN] Vanishing Gradient Problem

Batch Normalization Layer is one of the solutions to mitigate this problem that the values are distributed across extremes. As shown in figure 4 above, Batch Normalization Layer performs normalization* on the tensor data obtained by some operations such as multiplication and addition. With this normalization, the gradients of activation function can be well-distributed (without zero-biased gradients) and it will lead to efficient parameter updates.