Survey on Attention-based Models Applied in NLP

Attention-based models are firstly proposed in the field of computer vision around mid 20141 (thanks for the remindar from @archychu). And then they spread into Natural Language Processing. In this post, I will mainly focus on a list of attention-based models applied in natural language processing. The first one should be the "guy" innovately and successfully bringing in attention mechanism from computer vision in to NLP. Although called RNNsearch by themselves in this paper, the model is in nature and inspired by the attention-based model as what they said in Section 3.1: Intuitively, this implements a mechanism of attention in the decoder.