From word embeddings to contextual word embeddings and Transfer Learning for NLP

Over the last couple of years, powerful deep learning methods have emerged to build industrial scale natural language understanding applications. The first wave of deep learning models employed pre-trained word embeddings (word2vec or GloVe) to initialize the first layer of a neural network followed by a task specific model trained using labelled data. The next wave of deep learning architectures (ELMo, ULMFiT, BERT) showed how to learn contextual word embeddings from massive amounts of unlabelled text data and then transfer this information to a wide variety of downstream tasks such as sentiment analysis, question answering etc. with limited amounts of labelled data. This approach is quite relevant for industrial settings where obtaining large amounts of labelled data is expensive. In this hands on tutorial, we will cover the important concepts behind recent developments such as word embeddings, sequence to sequence models, attention mechanism, contextual word embeddings, transfer learning and probing embeddings.