More Room for Language: Investigating the Effect of Retrieval on Language Models