On robust overfitting: adversarial training induced distribution matters

Despite their outstanding performance, deep neural networks (DNNs) are known to be vulnerable to adversarial attacks where a carefully designed perturbation may cause the network to make a wrong prediction [1, 2]. Many methods have been proposed to improve the robustness of DNNs against adversarial perturbations [3, 4, 5], among which Projected Gradient Descend based Adversarial Training (PGD-AT) [3] is arguably the most effective [6, 7]. A recent work in [8] however revealed a surprising phenomenon in PGD-AT: after training, even though the robust error (i.e., error probability in the predicted label for adversarially perturbed instances) is nearly zero on the training set, it may remain very high on the testing set. For example, on the testing set of CIFAR10, the robust error of PGD-AT trained model can be as large as 44.19%. This significantly contrasts the standard training: on CIFAR10, when the standard error (i.e., the error probability in the predicted label for non-perturbed instances) is nearly zero on the training set, its value on the testing set is only about 4%.