Decoding NLP Attention Mechanisms

Arguably more famous today than Michael Bay's Transformers, the transformer architecture and transformer-based models have been breaking all kinds of state-of-the-art records. They are (rightfully) getting the attention of a big portion of the deep learning community and researchers in Natural Language Processing (NLP) since their introduction in 2017 by the Google Translation Team. This architecture has set the stage for today's heavy-weight models: Google AI's BERT (and its variants) have been sitting in first position across many NLP leaderboards. OpenAI's GPT2 was judged so powerful by its authors, that up until recently only a weaker version of it was publicly released, following expressed concerns that this model might be used for "evil"! In this blogpost series, we will walk you through the rise of the transformer architecture.