VIbCReg: Variance-Invariance-better-Covariance Regularization for Self-Supervised Learning on Time Series

In the last year, representation learning (RL) has had great success within computer vision, improving both on SOTA for fine-tuned models and achieving close-to SOTA results on linear evaluation on the learned representations [1, 2, 3, 4, 5, 6, 7], and many more. The main idea in these papers is to train a high-capacity neural network using a self-supervised learning (SSL) loss that is able to produce representations of images that are useful for downstream tasks such as image classification and segmentation. The recent mainstream SSL frameworks can be divided into two main categories: 1) contrastive learning method, 2) non-contrastive learning method. The representative contrastive learning methods such as MoCo [3] and SimCLR [8] use positive and negative pairs and they learn representations by pulling the representations of the positive pairs together and pushing those of the negative pairs apart. However, these methods require a large number of negative pairs per positive pair to learn representations effectively.