Batch Normalization is a Cause of Adversarial Vulnerability

Batch norm is a standard component of modern deep neural networks, and tends to make the training process less sensitive to the choice of hyperparameters in many cases [13]. While ease of training is desirable for model developers, an important concern among stakeholders is that of model robustness to plausible, previously unseen inputs during deployment. The adversarial examples phenomenon has exposed unstable predictions across state-of-the-art models [27]. This has led to a variety of methods that aim to improve robustness, but doing so effectively remains a challenge [1, 20, 11, 14]. We believe that a prerequisite to developing methods that increase robustness is an understanding of factors that reduce it. Approaches for improving robustness often begin with existing neural network architectures--that use batch norm--and patching them against specific attacks, e.g.,