Self-Supervised Learning And Its Applications - AI Summary

The focus was largely on supervised learning methods that require huge amounts of labeled data to train systems for specific use cases. Bidirectional Encoder Representations from Transformers (BERT) a paper published by researchers at the Google AI team has become a gold standard when it comes to several NLP tasks such as Natural Language Inference (MNLI), Question Answering (SQuAD), and more. To make BERT handle a variety of downstream tasks, input representation is able to unambiguously represent a pair of sentences that are packed together in a single sequence. While autoencoding models like BERT utilize self-supervised learning for tasks like sentence classification (next or not), another application of self-supervised approaches lies in the domain of text generation. The inputs are passed through our pre-trained model to obtain the final transformer block's activation hm l, which is then fed into an added linear output layer with parameters W y to predict y: Translation Language Modelling (TLM): a new addition and an extension of MLM, where instead of considering monolingual text streams, parallel sentences are concatenated as illustrated in the following image.