Pixel Sentence Representation Learning

Vanilla language models are long known to have subpar sentence-level representation (Reimers and Gurevych, 2019; Wang et al., 2023), even worse than averaging static word embeddings (Pennington et al., 2014), i.e., sentence representations attained by pooling from sub-word embeddings encoded by language models do not closely reflect the relative semantics of sentences. Encouraged by the remarkable success of visual representation learning facilitated by unsupervised contrastive learning (Chen et al., 2020; He et al., 2020), efforts in NLP are made to leverage unsupervised contrastive learning to recover sentence-level encoding abilities from the models (Fang et al., 2020; Wu et al., 2020; Gao et al., 2021; Meng et al., 2021). However, translating the advancements in visual representation learning to learning sentence-level textual semantics presents unique challenges: a single augmentation (Wu et al., 2020; Meng et al., 2021) might alter the meaning of a sentence, posing problems of the validity of the augmented sentence as a positive pair. Such attempts are primarily bottlenecked by the discreteness of subword units brought by tokenization (Sennrich et al., 2016), impeding the creation of continuous unsupervised semantic pairs that have preserved semantics through small perturbations to inputs. Therefore, the most recognized unsupervised sentence representation learning method in NLP applies two dropout masks to the identical input to attain two representations, as positive pairs in contrastive learning (Gao et al., 2021). We argue that using identical inputs confines the method of Gao et al. (2021) to essentially only a way to improve uniformity (Wang and Isola, 2020) by distancing negative examples that are not identical to an instance itself, lacking