LLM Trainer: Automated Robotic Data Generating via Demonstration Augmentation using LLMs

Abstract-- We present LLM Trainer, a fully automated pipeline that leverages the world knowledge of Large Language Models (LLMs) to transform a small number of human demonstrations (as few as one) into a large robot dataset for imitation learning. Our approach decomposes demonstration generation into two steps: (1) offline demonstration annotation that extracts keyframes, salient objects, and pose-object relations; and (2) online keypose retargeting that adapts those keyframes to a new scene, given an initial observation. Using these modified keypoints, our system warps the original demonstration to generate a new trajectory, which is then executed, and the resulting demo, if successful, is saved. Because the annotation is reusable across scenes, we use Thompson sampling to optimize the annotation, significantly improving generation success rate. We evaluate our method on a range of tasks, and find that our data annotation method consistently outperforms expert-engineered baselines. We further show an ensemble policy that combines the optimized LLM feed-forward plan with a learned feedback imitation learning controller . Finally, we demonstrate hardware feasibility on a Franka Emika Panda robot. Recent advances in Large Language Models (LLMs) have revolutionized the field of robot learning, with applications ranging from task planning [1], to tool use in long horizon tasks [2], to deformable object manipulation [3]. At the core of these works is the LLM's broad base of world knowledge, gathered from training on internet-scale data, which allows these agents to be extremely generalizable. In this work, we seek to leverage the world knowledge of LLMs to fully automate demonstration generation through human demo augmentation.