FROM Pre-trained Word Embeddings TO Pre-trained Language Models -- Focus on BERT

Language modeling is the task of assigning a probability distribution over sequences of words that matches the distribution of a language. Although it sounds formidable, language modeling (i.e. ELMo, BERT, GPT) is essentially just predicting words in a blank. More formally, given a context, a language model predicts the probability of a word occurring in that context. Why is this method effective?