LangWBC: Language-directed Humanoid Whole-Body Control via End-to-end Learning

Figure 1: We propose a language-directed humanoid whole-body control framework that translates natural language commands into continuous robot actions through a Conditional V ariational Autoencoder (CV AE). The structured latent space brought by the CV AE enables smooth transitions between diverse and agile behaviors, as shown in the sequence where the robot seamlessly transitions from walking to running, concluding with a hand-waving motion prompted by the corresponding text commands. See more experiments at https://youtu.be/9AN0GulqWwc Abstract --General-purpose humanoid robots are expected to interact intuitively with humans, enabling seamless integration into daily life. Natural language provides the most accessible medium for this purpose. However, translating language into humanoid whole-body motion remains a significant challenge, primarily due to the gap between linguistic understanding and physical actions. In this work, we present an end-to-end, language-directed policy for real-world humanoid whole-body control. Our approach combines reinforcement learning with policy distillation, allowing a single neural network to interpret language commands and execute corresponding physical actions directly. T o enhance motion diversity and compositionality, we incorporate a Conditional V ariational Autoencoder (CV AE) structure. The resulting policy achieves agile and versatile whole-body behaviors conditioned on language inputs, with smooth transitions between various motions, enabling adaptation to linguistic variations and the emergence of novel motions. Please see our website at Lang-WBC.github.io