Lipschitz regularity of deep neural networks: analysis and efficient estimation

Deep neural networks made a striking entree in machine learning and quickly became state-of-the-art algorithms in many tasks such as computer vision [1, 2, 3, 4], speech recognition and generation [5, 6] or natural language processing [7, 8]. However, deep neural networks are known for being very sensitive to their input, and adversarial examples provide a good illustration of their lack of robustness [9, 10]. Indeed, a well-chosen small perturbation of the input image can mislead a neural network and significantly decrease its classification accuracy. One metric to assess the robustness of neural networks to small perturbations is the Lipschitz constant (see Definition 1), which upper bounds the relationship between input perturbation and output variation for a given distance. For generative models, the recent Wasserstein GAN [11] improved the training stability of GANs by reformulating the optimization problem as a minimization of the Wasserstein distance between the real and generated distributions [12]. However, this method relies on an efficient way of constraining the Lipschitz constant of the critic, which was only partially addressed in the original paper, and the object of several followup works [13, 14]. Recently, the Lipschitz continuity was used in order to improve the state-of-the-art in several deep 1 learning topics: (1) for robust learning, avoiding adversarial attacks was achieved in [15] by constraining local Lipschitz constants in neural networks.