DSDrive: Distilling Large Language Model for Lightweight End-to-End Autonomous Driving with Unified Reasoning and Planning

--In conventional end-to-end frameworks for Autonomous driving (AD), the underlying cognitive processes are inadequately addressed. While large language models (LLMs) offer improved understanding and reasoning capabilities, integrating them into AD systems poses two major challenges. First, there is an invariably significant discrepancy between the high computational demands of LLMs and the high efficiency required for autonomous vehicles. Second, although LLMs can generate high-quality semantic reasoning, it remains an open challenge to map high-level textual reasoning to low-level trajectory planning for autonomous vehicles. T o deal with these issues, we present DSDrive, a streamlined end-to-end paradigm tailored for integrating the reasoning and planning of autonomous vehicles into a unified framework. DSDrive leverages a compact LLM that employs a distillation method to preserve the enhanced reasoning capabilities of a larger-sized vision language model (VLM). T o effectively align the reasoning and planning tasks, a waypoint-driven dual-head coordination module is further developed, which synchronizes dataset structures, optimization objectives, and the learning process. DSDrive has been thoroughly tested in closed-loop simulations, where it performs on par with benchmark models and even outperforms in many key metrics, all while being more compact in size. Additionally, the computational efficiency of DSDrive (as reflected in its time and memory requirements during inference) has been significantly enhanced. Evidently thus, this work brings promising aspects and underscores the potential of lightweight systems in delivering interpretable and efficient solutions for AD. Video of the experiments: https://youtu.be/op8PzQurugY Autonomous vehicles (A Vs) have demonstrated pervasive power in modern intelligent transportation systems, which significantly enhances on-road driving safety and travel efficiency [1]. The end-to-end (E2E) autonomous driving (AD) framework [2], leveraging large-scale driving datasets [3] and advanced learning methodologies [4], plays an important role in advancing the forefront of A Vs. This fully differentiable framework processes raw sensor inputs to output low-level control through joint task optimization. Wenru Liu, Pei Liu, and Jun Ma are with The Hong Kong University of Science and Technology, China (e-mail: wliu354@connect.hkust-gz.edu.cn;