Lipschitz-Margin Training: Scalable Certification of Perturbation Invariance for Deep Neural Networks

This indicates that even protected networks can be unexpectedly vulnerable. This is a crucial problem for this specific line of research because the primary concern of these studies are security threats. To tackle this crucial problem, we aim to develop defense methods with theoretical guarantees. Our goal is to ensure the lower bounds on the size of adversarial perturbations that networks can never be deceived for each input. We refer to these lower bounds as certified invariant radii, or simply, invariant radii. To make them available in broad applications, there are two fundamental requirements to their calculation methods: 1. the minimality of assumptions on network structures, 2. the computational tractability. However, many existing approaches require strong assumptions and massive computational costs. For example, we could not ensure perturbation invariance for some network structures such as wide residual networks [42], which have been commonly used in the evaluations of defense methods. This work tackled this problem and we provide a widely applicable, yet, highly scalable method to ensure large invariant radii. Our basic idea is to bound the size of adversarial perturbations that networks can never be deceived Even though the concept of using the Lipschitz constant has already appeared in Szegedy et al. [37], how much certifications they can provide has not been studied well. We show we can ensure significantly larger invariant radii compared to a recent computationally efficient counterpart [32]. However, the size of certified invariant radii can still be insufficient to be practically meaningful in some cases. We addressed this issue with a novel training procedure that further strengthen perturbation invariance.