S2AP: Score-space Sharpness Minimization for Adversarial Pruning

Adversarial pruning methods have emerged as a powerful tool for compressing neural networks while preserving robustness against adversarial attacks. These methods typically follow a three-step pipeline: (i) pretrain a robust model, (ii) select a binary mask for weight pruning, and (iii) finetune the pruned model. To select the binary mask, these methods minimize a robust loss by assigning an importance score to each weight, and then keep the weights with the highest scores. However, this score-space optimization can lead to sharp local minima in the robust loss landscape and, in turn, to an unstable mask selection, reducing the robustness of adversarial pruning methods. To overcome this issue, we propose a novel plug-in method for adversarial pruning, termed Score-space Sharpness-aware Adversarial Pruning (S2AP). Through our method, we introduce the concept of score-space sharpness minimization, which operates during the mask search by perturbing importance scores and minimizing the corresponding robust loss. Extensive experiments across various datasets, models, and sparsity levels demonstrate that S2AP effectively minimizes sharpness in score space, stabilizing the mask selection, and ultimately improving the robustness of adversarial pruning methods. Deep neural networks are susceptible to adversarial attacks, which entail optimizing an input perturbation added to the original sample to induce a misclassification (Biggio et al., 2013; Szegedy et al., 2014). Besides robustness against adversarial examples, networks are often required to be compact and suitable for resource-constrained scenarios (Liu & Wang, 2023), where the model's dimension cannot be chosen at hand but requires respecting a given constraint. In this regard, neural network pruning (LeCun et al., 1989) represents a powerful compression method by removing redundant or less impactful parameters according to a desired sparsity rate and, as a result, allowing the preservation of much of the performance of a dense model counterpart (Blalock et al., 2020). Adversarial Pruning (AP) methods aim to fulfill this twofold requirement, thus extending model compression to the adversarial case, by removing parameters less responsible for adversarial robustness drops (Piras et al., 2024).