SenseRAG: Constructing Environmental Knowledge Bases with Proactive Querying for LLM-Based Autonomous Driving

LLMs possess a significant advantage in their ability to recognize environmental information, which enables the This study addresses the critical need for enhanced situational system to handle the complex environment [6]. Unlike other awareness in autonomous driving (AD) by leveraging perception technologies, LLMs can truly "understand" the the contextual reasoning capabilities of large language context [7], while models like computer vision (CV) rely on models (LLMs). Unlike traditional perception systems that rigid, predefined labels learned during training. CV models rely on rigid, label-based annotations, it integrates realtime, are constrained by fixed annotations and lack flexibility in multimodal sensor data into a unified, LLMs-readable new scenarios [8]. In contrast, LLMs can dynamically process knowledge base, enabling LLMs to dynamically understand diverse contexts and relationships within data. However, and respond to complex driving environments. To overcome their main limitation is that they are designed to handle the inherent latency and modality limitations of LLMs, language-based information and cannot directly process a proactive Retrieval-Augmented Generation (RAG) is designed the multimodal sensor data from Vehicle to Anything (V2X) for AD, combined with a chain-of-thought prompting and AD systems, such as radar, cameras, or Lidar [9] [10].