Improving embedding with contrastive fine-tuning on small datasets with expert-augmented scores

Text embedding models are fundamental in natural language processing (NLP), serving as lowdimensional vector representations that capture semantic similarity between texts (Aggarwal & Zhai, 2012; Angelov, 2020). They are critical for tasks such as text classification, retrieval, question answering, and dialogue systems. Recent advancements in large language models (LLMs) have spurred interest in retrieval-augmented systems that integrate LLM reasoning with the efficiency of text embeddings. Two main research directions exist: enhancing performance in semantic textual similarity (STS) through supervised fine-tuning, normalization, and unsupervised contrastive learning; and addressing text retrieval through dual-encoder architectures and self-supervised pre-training (Li et al., 2023; Wang et al., 2022; Izacard & Grave, 2020; Ren et al., 2021;; Devlin, 2018; Vaswani, 2017). Recent studies have aimed to create unified text representation models through large-scale contrastive learning and prompt-based learning (Muennighoff, 2022), evaluated on benchmarks like the massive text embedding benchmark (MTEB) (Muennighoff et al., 2022), which assesses models across 56 datasets and seven categories.