Can We Break Free from Strong Data Augmentations in Self-Supervised Learning?

Self-supervised learning (SSL) has emerged as a promising solution for addressing the challenge of limited labeled data in deep neural networks (DNNs), offering scalability potential. However, the impact of design dependencies within the SSL framework remains insufficiently investigated. In this study, we comprehensively explore SSL behavior across a spectrum of augmentations, revealing their crucial role in shaping SSL model performance and learning mechanisms. Leveraging these insights, we propose a novel learning approach that integrates prior knowledge, with the aim of curtailing the need for extensive data augmentations and thereby amplifying the efficacy of learned representations. Notably, our findings underscore that SSL models imbued with prior knowledge exhibit reduced texture bias, diminished reliance on shortcuts and augmentations, and improved robustness against both natural and adversarial corruptions. These findings not only illuminate a new direction in SSL research, but also pave the way for enhancing DNN performance while concurrently alleviating the imperative for intensive data augmentation, thereby enhancing scalability and realworld problem-solving capabilities. Deep neural networks (DNNs) have proven to be highly effective in encoding patterns in data distribution to produce powerful and rich representations that have improved generalization performance across various perception tasks, such as classification, detection, and segmentation. However, one of the major limitations is that DNNs are data-hungry and annotating millions of available data is expensive. Self-supervised learning (SSL) has been proposed as a promising solution to this issue, to enable the learning of useful representations without manual annotations. Self-supervised learning paradigm needs to ensure that the resulting features are generic to be applicable to a wide range of real-world applications. Various SSL methods, including pretext-based (Gidaris Figure 1: The impact of augmentations on SSL methods is et al., 2018; Noroozi & Favaro, 2016), critical: as removing strong augmentations from SSL training contrastive-based (Chen et al., 2020a; He et al., can result in a significant drop in their performance.