Learning from Risk: LLM-Guided Generation of Safety-Critical Scenarios with Prior Knowledge

Autonomous driving faces critical challenges in rare long-tail events and complex multi-agent interactions, which are scarce in real-world data yet essential for robust safety validation. This paper presents a high-fidelity scenario generation framework that integrates a conditional variational autoencoder (CVAE) with a large language model (LLM). The CVAE encodes historical trajectories and map information from large-scale naturalistic datasets to learn latent traffic structures, enabling the generation of physically consistent base scenarios. This knowledge-driven optimization balances realism with controllability, ensuring that generated scenarios remain both plausible and risk-sensitive. Extensive experiments in CARLA and SMARTS demonstrate that our framework substantially increases the coverage of high-risk and long-tail events, improves consistency between simulated and real-world traffic distributions, and exposes autonomous driving systems to interactions that are significantly more challenging than those produced by existing rule-or data-driven methods. These results establish a new pathway for safety validation, enabling principled stress-testing of autonomous systems under rare but consequential events. Introduction The safety and reliability of autonomous driving depend on rigorous validation under diverse test conditions, especially in high-risk, highly interactive, and safety-critical scenarios (Wang et al., 2021; Hossain, 2025). Yet such events are extremely scarce in real-world datasets, creating a persistent gap between development testing and deployment needs. Simulation-based methods provide an effective alternative by generating large numbers of rare and adversarial environments, thereby alleviating data scarcity and enabling controlled safety evaluation (Huang et al., 2020). To address these challenges, this paper proposes a risk knowledge-guided traffic scene generation framework that integrates a Conditional Variational Autoencoder (CV AE) with a Large Language Model (LLM). Unlike prior works that merely sample or replay specific risky cases, the proposed framework establishes a general and controllable pipeline for synthesizing diverse safety-critical scenarios under varying risk conditions. The CVAE learns latent spatiotemporal representations from real-world trajectories and maps to generate physically coherent base scenes, while the LLM acts as a knowledge-driven controller that interprets scene semantics, analyzes multi-agent risk interactions, and dynamically adjusts optimization objectives to guide the generation toward desired levels of behavioral complexity and risk exposure.