Curriculum learning for language modeling

Seeking to represent natural language, researchers have found language models (LM) with Sesame Street-inspired names [1] [2] [3] to be incredibly effective methods of producing language representations (LR). These LM's have leverage transfer learning by training on a large text corpus to learn a good representation of language which can then be used in a down steam task like Question Answering or Entity Resolution. While these LMs have shown to be excellent methods to enable language understanding, the ability to train these models is becoming increasingly computationally expensive [4]. Since model performance is closely tied to the size of training data, model size, and compute used to train [5] the bulk of existing research has focused on scaling these aspects without much focus on increasing efficiency of training. Seeking to explore what methods could be used to make LM training more efficient we study the effect of curriculum learning by training ELMo with a wide variety of curricula.