Unsupervised Representation Transfer for Small Networks: I Believe I Can Distill On-the-Fly

Notably, our ResNet-18, trained with ResNet-50 teacher, achieves 68.3% ImageNet Top-1 accuracy on frozen feature linear evaluation, which is only 1.5% below the