A Limitations and Potential Negative Social Impacts

Our work investigates the "larger teacher, worse student" phenomena in knowledge However, we only discuss image classification. Therefore, we do not guarantee the validity of our observation on other tasks, i.e., object detection, In addition, these classes can be sensitive, i.e., gender We hope future work can completely resolve this issue. Since most of these method provides hyper-parameters for CIFAR100, we do not modify them. In Section 2.2 we use modified ResNet24 as student to perform KD on a ResNet56 teacher model. We have mentioned the existence of the undistillable classes in general to various methods, and Table 1 gives a comprehensive list of methods for which we studied.