Scientists have made tremendous leaps in getting computers to understand natural language, as well as in generating a series of physical poses to create realistic animations. These capabilities might as well exist in separate worlds, however, because the link between natural language and physical poses has been missing. Louis-Philippe Morency, associate professor in the Language Technologies Institute (LTI), and Chaitanya Ahuja, an LTI Ph.D. student, are working to bring those worlds together using a neural architecture they call Joint Language-to-Pose, or JL2P. The JL2P model enables sentences and physical motions to be jointly embedded, so it can learn how language is related to action, gestures and movement. "I think we're in an early stage of this research, but from a modeling, artificial intelligence and theory perspective, it's a very exciting moment," Morency said.

Carnegie Mellon University AI researchers have created an AI agent that is able to translate words into physical movement. Called Joint Language-to-Pose, or JL2P, the approach combines natural language with 3D pose models. The pose forecasting joint embedding is trained with end-to-end curriculum learning, an approach that stresses shorter task completion sequences before moving on to harder objectives. JL2P animations are limited to stick figures today, but the ability to translate words into human-like movement can someday help humanoid robots do physical tasks in the real world or assist creatives in animating virtual characters for things like video games or movies. JL2P is in line with previous works that turn words into imagery -- like Microsoft's ObjGAN, which sketches images and storyboards from captions, Disney's AI that uses words in a script to create storyboards, and Nvidia's GauGAN, which lets users paint landscapes using paintbrushes labeled with words like "trees," "mountain," or "sky."