Should You Use Your Large Language Model to Explore or Exploit?