Our dataset is released under the CC BY -NC 4.0 license, allowing We include more dataset visualizations depicting various ego-actions in Figure 9 . Interaction plays a crucial role in driving scenarios. The sixth scenario showcases encountering road construction ahead, followed by encountering a street sweeper in the seventh scenario. Typically belonging to the tail end of a long-tail distribution, these scenarios are rare yet crucial for ensuring safe driving. Lane Changing: Changing lanes to overtake slower vehicles or merge into traffic.

Driving world models have gained increasing attention due to their ability to model complex physical dynamics. However, their superb modeling capability is yet to be fully unleashed due to the limited video diversity in current driving datasets. We introduce DrivingDojo, the first dataset tailor-made for training interactive world models with complex driving dynamics. We further define an action instruction following (AIF) benchmark for world models and demonstrate the superiority of the proposed dataset for generating action-controlled future predictions.

Driving world models have gained increasing attention due to their ability to model complex physical dynamics. However, their superb modeling capability is yet to be fully unleashed due to the limited video diversity in current driving datasets. We introduce DrivingDojo, the first dataset tailor-made for training interactive world models with complex driving dynamics. Our dataset features video clips with a complete set of driving maneuvers, diverse multi-agent interplay, and rich open-world driving knowledge, laying a stepping stone for future world model development. We further define an action instruction following (AIF) benchmark for world models and demonstrate the superiority of the proposed dataset for generating action-controlled future predictions.