Self Supervised Representation Learning in NLP

While Computer Vision is making amazing progress on self-supervised learning only in the last few years, self-supervised learning has been a first-class citizen in NLP research for quite a while. Language Models have existed since the 90's even before the phrase "self-supervised learning" was termed. The Word2Vec paper from 2013 popularized this paradigm and the field has rapidly progressed applying these self-supervised methods across many problems. At the core of these self-supervised methods lies a framing called "pretext task" that allows us to use the data itself to generate labels and use supervised methods to solve unsupervised problems. These are also referred to as "auxiliary task" or "pre-training task". The representations learned by performing this task can be used as a starting point for our downstream supervised tasks.