On Retrieval Augmentation and the Limitations of Language Model Training