Assessing "Implicit" Retrieval Robustness of Large Language Models