Batch-Softmax Contrastive Loss for Pairwise Sentence Scoring Tasks

The use of contrastive loss for representation learning has become prominent in computer vision, and it is now getting attention in Natural Language Processing (NLP). Here, we explore the idea of using a batch-softmax contrastive loss when fine-tuning large-scale pre-trained transformer models to learn better task-specific sentence embeddings for pairwise sentence scoring tasks. We introduce and study a number of variations in the calculation of the loss as well as in the overall training procedure; in particular, we find that data shuffling can be quite important. Our experimental results show sizable improvements on a number of datasets and pairwise sentence scoring tasks including classification, ranking, and regression. Finally, we offer detailed analysis and discussion, which should be useful for researchers aiming to explore the utility of contrastive loss in NLP. Recent years have seen a revolution in Natural Language Processing (NLP) thanks to the advances in machine learning. While a lot of attention has been paid to the architectures, especially for deep learning, there has been less focus on studying loss functions. At the same time, loss functions based on similar or on the same ideas were reinvented multiple times under different names. This can cause difficulties when solving new problems or when designing new experiments based on previous results. To a greater extent, this applies to "universal" loss functions, which can be applied in different machine learning areas and tasks such as Computer Vision (CV), Recommendation Systems, and NLP.