DriveE2E: Closed-Loop Benchmark for End-to-End Autonomous Driving through Real-to-Simulation

Closed-loop evaluation is increasingly critical for end-to-end autonomous driving. Current closed-loop benchmarks using the CARLA simulator rely on manually configured traffic scenarios, which can diverge from real-world conditions, limiting their ability to reflect actual driving performance. To address these limitations, we introduce a simple yet challenging closed-loop evaluation framework that closely integrates real-world driving scenarios into the CARLA simulator with infrastructure cooperation. Our approach involves extracting 800 dynamic traffic scenarios selected from a comprehensive 100-hour video dataset captured by high-mounted infrastructure sensors, and creating static digital twin assets for 15 real-world intersections with consistent visual appearance. These digital twins accurately replicate the traffic and environmental characteristics of their real-world counterparts, enabling more realistic simulations in CARLA. This evaluation is challenging due to the diversity of driving behaviors, locations, weather conditions, and times of day at complex urban intersections. In addition, we provide a comprehensive closed-loop benchmark for evaluating end-to-end autonomous driving models. Red circle denotes the selected ego vehicle. End-to-End Autonomous Driving (E2EAD) has shown great advances and potential. Effective evaluation is essential for assessing the driving capabilities of E2EAD models, thereby advancing research and promoting the development of improved algorithms. Traditionally, E2EAD performance has been assessed using open-loop evaluation, which operates on prerecorded expert driving trajectories and corresponding sensor data, as seen in datasets such as nuScenes Caesar et al. (2020). In this setting, the model passively predicts actions without influencing future observations, making the task resemble trajectory prediction Zhai et al. (2023); Li et al. (2024b). As a result, open-loop evaluation provides limited insight into vehicle-environment interactions and real-time decision-making. In contrast, closed-loop evaluation continuously updates observations based on the ego vehicle's actions, allowing the E2EAD model to control the vehicle using its own decisions.