Comprehensive Knowledge Distillation with Causal Intervention

On the other hand, as the teacher has learned rich and powerful representations, the existing approaches enforce the student to fully mimic the behavior of the teacher.