One of the latest self-supervised learning (SSL) methods, VICReg, showed a great performance both in the linear evaluation and the fine-tuning evaluation. However, VICReg is proposed in computer vision and it learns by pulling representations of random crops of an image while maintaining the representation space by the variance and covariance loss. However, VICReg would be ineffective on non-stationary time series where different parts/crops of input should be differently encoded to consider the non-stationarity. Another recent SSL proposal, Temporal Neighborhood Coding (TNC) is effective for encoding non-stationary time series. This study shows that a combination of a VICReg-style method and TNC is very effective for SSL on non-stationary time series, where a non-stationary seismic signal time series is used as an evaluation dataset.

In the last year, representation learning (RL) has had great success within computer vision, improving both on SOTA for fine-tuned models and achieving close-to SOTA results on linear evaluation on the learned representations [1, 2, 3, 4, 5, 6, 7], and many more. The main idea in these papers is to train a high-capacity neural network using a self-supervised learning (SSL) loss that is able to produce representations of images that are useful for downstream tasks such as image classification and segmentation. The recent mainstream SSL frameworks can be divided into two main categories: 1) contrastive learning method, 2) non-contrastive learning method. The representative contrastive learning methods such as MoCo [3] and SimCLR [8] use positive and negative pairs and they learn representations by pulling the representations of the positive pairs together and pushing those of the negative pairs apart. However, these methods require a large number of negative pairs per positive pair to learn representations effectively.