Learning to Generate Better Than Your LLM