A Measuring Diffused Redundancy

A.1 CKA Definition In all our evaluations we use CKA with a linear kernel [24] which essentially amounts to the following steps: 1. A.2 Additional CKA results Fig 9 shows CKA comparison between randomly chosen parts of the layer and the full layer for different kinds of ResNet50. We observe that even ResNet50 trained with MRL loss shows a significant amount of diffused redundancy. Figure 9: [Comparison of Diffused Redundancy in MRL vs other losses, through the lens of CKA] We see a similar trend as reported in Fig 7 in the main paper, where even the MRL model shows a significant amount of diffused redundancy despite being explicitly trained to instead have structured redundancy. The amount of diffused redundancy however is much lesser than the resnets trained using the standard loss and adv. Here we list the sources of weights for the various pre-trained models used in our experiments: ResNet18 trained on ImageNet1k using standard loss: taken from timm v0.6.1.