Tanh Works Better With Asymmetry

Neural Information Processing Systems 

Batch Normalization is commonly located in front of activation functions, as proposed by the original paper.