A refresher on batch (re-)normalization – Luminovo – Medium

When the mini-batch mean (µB) and mini-batch standard deviation (σB) diverge from the mean and standard deviation over the entire training set too often, BatchNorm breaks. Remember that at inference time we use the moving averages of µB and σB (as an estimate of the statistics of the entire training set) to do the normalization step. Naturally, if your means and standard deviations during training and testing are different, so are your activations and you can't be surprised if your results are different (read worse), too. This can happen when your mini-batch samples are non-i.i.d.