Knowledge distillation aims to transfer useful information from a teacher network to a student network, with the primary goal of improving the student's performance for the task at hand. Over the years, there has a been a deluge of novel techniques and use cases of knowledge distillation. Yet, despite the various improvements, there seems to be a glaring gap in the community's fundamental understanding of the process. Specifically, what is the knowledge that gets distilled in knowledge distillation? In other words, in what ways does the student become similar to the teacher?

MIM to enhance the adversarial robustness of downstream models. It is important to highlight that our paper's focus is specifically on the adversarial robustness of ViTs. It is shown that our method can provide an effective defense against severe adversarial attacks. We propose two hypotheses for explaining the reason behind our method's effectiveness: (1) Given Figure 3 (a) shows the comparison between the results of noise being known and unknown. When the attacker can access the noise, our model's robust accuracy does not improve much as The results indicate that both proposed hypotheses are true.

Neural Information Processing Systems http://nips.cc/

While many techniques for detecting these attacks have been proposed, theyareeasily bypassed when theadversary hasfullknowledge of the detection mechanism and adapts the attack strategy accordingly. In this paper,we adopt anovel perspectiveand regard the omnipresence of adversarial perturbations asastrength rather thanaweakness.

Deep networks achieve impressive accuracy and wide adoption in computer vision [17], speech recognition [14], and natural language processing [21].

Neural Information Processing Systems http://nips.cc/