Non-myopic Generation of Language Models for Reasoning and Planning

Large Language Models (LLMs) have demonstrated remarkable abilities in reasoning and planning by breaking down complex problems into sequential steps. This paper revisits LLM reasoning from an optimal control perspective, proposing a novel method, Predictive-Decoding, that leverages Model Predictive Control to enhance planning accuracy. By reweighting LLM distributions based on foresight trajectories, Predictive-Decoding aims to mitigate early errors and promote non-myopic planning. Our experiments show significant improvements across a wide range of tasks in math, coding, and agent-based scenarios. Furthermore, Predictive-Decoding demonstrates computational efficiency, outperforming search baselines while utilizing inference compute more effectively. This study provides insights into optimizing LLM planning capabilities. Code is available at this repo. Large Language Models (LLMs) are extensively pretrained on large corpus to predict the next tokens.