LanguageMPC: Large Language Models as Decision Makers for Autonomous Driving

Existing learning-based autonomous driving (AD) systems face challenges in comprehending high-level information, generalizing to rare events, and providing interpretability. To address these problems, this work employs Large Language Models (LLMs) as a decision-making component for complex AD scenarios that require human commonsense understanding. We devise cognitive pathways to enable comprehensive reasoning with LLMs, and develop algorithms for translating LLM decisions into actionable driving commands. Through this approach, LLM decisions are seamlessly integrated with low-level controllers by guided parameter matrix adaptation. Extensive experiments demonstrate that our proposed method not only consistently surpasses baseline approaches in single-vehicle tasks, but also helps handle complex driving behaviors even multi-vehicle coordination, thanks to the commonsense reasoning capabilities of LLMs. This paper presents an initial step toward leveraging LLMs as effective decision-makers for intricate AD scenarios in terms of safety, efficiency, generalizability, and interoperability. We aspire for it to serve as inspiration for future research in this field. Imagine you are behind the wheel, approaching an unsignalized intersection and planning to turn left, with an oncoming vehicle straight ahead. Human drivers intuitively know that according to traffic rules, they should slow down and yield, even if it is technically possible to speed through. However, existing advanced learning-based Autonomous Driving (AD) systems typically require complex rules or reward function designs to handle such scenarios effectively (Chen et al., 2023a; Kiran et al., 2022). This reliance on predefined rule bases often limits their ability to generalize to various situations. Another challenge facing existing learning-based AD systems is the long-tail problem (Buhet et al., 2019). Both limited datasets and sampling efficiency (Atakishiyev et al., 2023) can present challenges for existing learning-based AD systems when making decisions in rare real-world driving scenarios. Chauffeurnet (Bansal et al., 2018) demonstrated such limits where even 30 million stateaction samples were insufficient to learn an optimal policy that mapped bird's-eye view images (states) to control (action).