What's the Plan? Evaluating and Developing Planning-Aware Techniques for Language Models

Planning is a fundamental task in artificial intelligence that involves finding a sequence of actions that achieve a specified goal in a given environment. Large language models (LLMs) are increasingly employed in applications that require such planning capabilities, including web and embodied agents. In line with recent studies, we demonstrate through experimentation that LLMs lack necessary skills required for planning. We focus on their ability to function as world models, and show that they struggle to simulate the complex dynamics of classic planning domains. Based on these observations, we advocate for the potential of a hybrid approach that combines language models with classical planning methodology. We introduce SimP lan, a novel hybrid architecture, utilizing external world modeling tools and the greedy best-first search algorithm. We assess its effectiveness in a rigorous set of experiments across a variety of challenging planning domains. Our results demonstrate that SimP lan significantly outperforms existing LLM-based planners, highlighting the critical role of search strategies and world models in planning applications.