Overview of Transfer Learning in NLP -- Contextual Word Vectors

Machine translation data can be the potential equivalent to ImageNet in NLP. The authors try to prove this hypothesis by adopting the attentional sequence-to-sequence model trained for machine translation. These models usually contain an LSTM-based encoder. They make use of this encoder to obtain, what they call, the contextual vectors. This process involves two steps.