However, the existing DL models are highly vulnerable to adversarial examples [55, 20], which are maliciously generated by an adversary to make a model produce erroneous predictions. As DL models have been integrated into various security-sensitive applications ( e.g., autonomous driving, healthcare, and finance), the study of the adversarial robustness issue has attracted increasing attention with an enormous number of adversarial attack and defense methods proposed. Therefore, it is crucial to conduct correct and rigorous evaluations of these methods for understanding their pros and cons, comparing their performance, and providing insights for building new methods [6]. The research on adversarial robustness is faced with an "arms race " between attacks and defenses, i.e ., a defense method proposed to prevent the existing attacks was soon evaded by new attacks, and vice versa [7, 8, 23, 1, 57, 67]. For instance, defensive distillation [43] was proposed to improve adversarial robustness, but was later shown to be ineffective against a strong attack [8].