Adapting a World Model for Trajectory Following in a 3D Game

Mendonc a 1 T arun Gupta Darren Gehring 1 Sam Devlin 1 Sergio V alcarcel Macua 1 Raluca Stevenson 1 1 Microsoft Research 2 Queen Mary University of London 3 University of Oxford A BSTRACT Imitation learning is a powerful tool for training agents by leveraging expert knowledge, and being able to replicate a given trajectory is an integral part of it. In this study, we apply Inverse Dynamics Models (IDM) with different encoders and policy heads to trajectory following in a modern 3D video game - Bleeding Edge. Additionally, we investigate several future alignment strategies that address the distribution shift caused by the aleatoric uncertainty and imperfections of the agent. We measure both the trajectory deviation distance and the first significant deviation point between the reference and the agent's trajectory and show that the optimal configuration depends on the chosen setting. Our results show that in a diverse data setting, a GPT -style policy head with an encoder trained from scratch performs the best, DINOv2 encoder with the GPT -style policy head gives the best results in the low data regime, and both GPT -style and MLP-style policy heads had comparable results when pre-trained on a diverse setting and fine-tuned for a specific behaviour setting. 1 I NTRODUCTION Using video games as a testbed for game-playing agents has been a thoroughly studied area. Although imitation learning and reinforcement learning have been applied, most of these algorithms (Vinyals et al., 2019a; Berner et al., 2019; Wurman et al., 2022) focused on superhuman behaviour, rather than matching human play style. Research on human-like play primarily leverages imitation learning, where the most popular techniques revolve around learning from demonstration (Abbeel & Ng, 2004; Ho & Ermon, 2016) and learning from observations (Torabi et al., 2018a; Y ang et al., 2019). In this work, we use learning from demonstrations to replicate a recorded trajectory in a complex 3D video game. In simple environments, trajectory replication can often be achieved by directly replaying recorded actions.