Convert Language Model into a Value-based Strategic Planner