Beyond Memorization: The Challenge of Random Memory Access in Language Models