Why Data should be Normalized before Training a Neural Network

Among the best practices for training a Neural Network is to normalize your data to obtain a mean close to 0. Normalizing the data generally speeds up learning and leads to faster convergence. Also, the (logistic) sigmoid function is hardly ever used anymore as an activation function in hidden layers of Neural Networks, because the tanh function (among others) seems to be strictly superior. While this might not be immediately evident, there are very similar reasons for why this is the case. The tanh function is quite similar to the logistic sigmoid. The main difference, however, is that the tanh function outputs results between -1 and 1, while the sigmoid function outputs values that are between 0 and 1 -- therefore they are always positive.