Seeing the Whole in the Parts in Self-Supervised Representation Learning

Here, we propose a new way to model spatial co-occurrences by aligning local representations (before pooling) with a global image representation. We present CO-SSL, a family of instance discrimination methods and show that it outperforms previous methods on several datasets, including ImageNet-1K where it achieves 71.5% of Top-1 accuracy with 100 pre-training epochs. CO-SSL is also more robust to noise corruption, internal corruption, small adversarial attacks, and large training crop sizes. Our analysis further indicates that CO-SSL learns highly redundant local representations, which offers an explanation for its robustness. Overall, our work suggests that aligning local and global representations may be a powerful principle of unsupervised category learning. Recent self-supervised learning (SSL) approaches learn visual representations that perform well on diverse downstream tasks, including object categorization. These methods include instance discrimination (ID) (Chen et al., 2020a) and masked modeling (MM) (He et al., 2022; Bao et al., 2021) approaches. While ID methods train representations to be invariant over different small crops of augmented images, MM uses a partially masked image to predict the masked information. Existing analyses suggest that the best configurations involve discarding most of the information from the images (Tian et al., 2021; He et al., 2022; Assran et al., 2023). For instance, if there is an elephant trunk, the method may learn that there is likely also an elephant tusk in the image, and vice versa. Intuitively, this makes category recognition (an elephant) less sensitive to the exact visual features (trunk or tusk) present in an image. Spatial statistical learning is also a fundamental aspect of biological vision. A classic study investigated the ability of humans to extract spatial co-occurrences among features (Fiser & Aslin, 2001).