How Can LLM Guide RL? A Value-Based Approach