Do LLM Modules Generalize? A Study on Motion Generation for Autonomous Driving

In parallel with the recent breakthroughs in large language models [1, 2], Transformer-based approaches have also achieved widespread success in the domain of autonomous driving motion generation, including tasks such as trajectory prediction [3, 4, 5], traffic simulation [6, 7, 8], and ego-vehicle planning [9, 10, 11]. However, most prior works have focused on empirically transferring individual modules from LLMs to autonomous driving applications. In contrast, this paper takes a more holistic perspective to explore the striking similarities between LLMs and autonomous driving motion generation, while also highlighting the subtle differences that lie beneath these similarities. Autonomous driving motion generation refers to the task of generating future trajectories for specified agents under a set of constraints, based on contextual information from the environment and the historical states of the agents. Transformer-based models--particularly those following the GPT architecture--have achieved considerable success in this domain in recent years [8, 12, 13, 14]. Their typical workflows encompass only a subset of the overall modeling process, consisting of the following components: T okenizing: compressing contextual and motion information into a sequence of tokens that can later be decoded into trajectories; Positional Embedding: encoding spatial and temporal relationships among agents, between agents and the environment, and within each agent's trajectory; Pre-training: training the model on large-scale self-supervised datasets to learn gener-alizable motion generation capabilities; Post-training: applying fine-tuning techniques to enforce additional constraints, such as collision avoidance or human-like behavior; T est-time Computing: leveraging increased compute budgets to dynamically optimize model outputs without modifying model parameters during inference.