Supplementary Materials: An Investigation into Whitening Loss for Self-supervised Learning

In this section, we provide the details of the implementation and training protocol for the experiments on small and medium size datasets (including CIFAR-10, CIFAR-100, STL-10, Tiny-ImageNet and ImageNet-100). Our implementation is based on the released codebase of W-MSE [7] CIFAR-10 and CIFAR-100 [13], two small-scale datasets composed of 32 32 images with 10 and 100 classes, respectively. STL-10 [4], derived from ImageNet [6], with 96 96 resolution images and more than 100K training samples. Tiny ImageNet [14], a reduced version of ImageNet [6], composed of 200 classes with images scaled down to 64 64. The total number of images is: 100K (training) and 10K (testing).