Understanding Attention in Natural Language Processing with 3 Projects

In this blog post, I'll summarize my understanding of attention used in natural language processing (NLP). As a machine learning and NLP self-learner, when I initially got exposed to the idea of attention, I felt overwhelmed by its whole bunch of different variations and all the nitty-gritties involved in the implementations. Now, after reading articles, blogs and code, watching YouTube videos and also implementing it myself in several projects, I found it actually not that hard to understand when looking back. Hopefully by sharing what I learned along the journey, I could help some of those who are also going though that learning process, especially beginners like who I was a couple of months ago, speed up their progress and make it a bit more enjoyable. The concept of attention was firstly widely spread because of its use in the sequence-to-sequence (seq2seq) model for neural machine translation.