Disentangling Exploration of Large Language Models by Optimal Exploitation