Goto

Collaborating Authors

 motiongpt model



Appendix of MotionGPT

Neural Information Processing Systems

There are [frames] frames in the motion. What is the total duration of [motion]'s body movements in seconds? The motion lasts for [seconds] seconds.Caption-to-Length How many frames are expected for the motion that matches [caption]? The duration is estimated to be around [frames] frames. Given [caption], provide the anticipated second duration for the corresponding motion. The motion has a length of [seconds] seconds.Length-to-Caption What are some possible physical gestures that could be made in [frames] frames?


MotionGPT: Human Motion as a Foreign Language

Jiang, Biao, Chen, Xin, Liu, Wen, Yu, Jingyi, Yu, Gang, Chen, Tao

arXiv.org Artificial Intelligence

Though the advancement of pre-trained large language models unfolds, the exploration of building a unified model for language and other multi-modal data, such as motion, remains challenging and untouched so far. Fortunately, human motion displays a semantic coupling akin to human language, often perceived as a form of body language. By fusing language data with large-scale motion models, motion-language pre-training that can enhance the performance of motion-related tasks becomes feasible. Driven by this insight, we propose MotionGPT, a unified, versatile, and user-friendly motion-language model to handle multiple motion-relevant tasks. Specifically, we employ the discrete vector quantization for human motion and transfer 3D motion into motion tokens, similar to the generation process of word tokens. Building upon this "motion vocabulary", we perform language modeling on both motion and text in a unified manner, treating human motion as a specific language. Moreover, inspired by prompt learning, we pre-train MotionGPT with a mixture of motion-language data and fine-tune it on prompt-based question-and-answer tasks. Extensive experiments demonstrate that MotionGPT achieves state-of-the-art performances on multiple motion tasks including text-driven motion generation, motion captioning, motion prediction, and motion in-between.