Learning Physical Interaction Skills from Human Demonstrations

Learning physical interaction skills--such as dancing, handshaking, or sparring--remains a fundamental challenge for agents operating in human environments, particularly when the agent's morphology differs significantly from that of the demonstrator. Existing approaches often rely on handcrafted objectives or morphological similarity, limiting their capacity for generalization. Here, we introduce a framework that enables agents with diverse embodiments to learn whole-body interaction behaviors directly from human demonstrations. The framework extracts a compact, transferable representation of interaction dynamics--called the Embedded Interaction Graph (EIG)--which captures key spatiotemporal relationships between the interacting agents. This graph is then used as an imitation objective to train control policies in physics-based simulations, allowing the agent to generate motions that are both semantically meaningful and physically feasible. We demonstrate BuddyImitation on multiple agents, such as humans, quadrupedal robots with manipulators, or mobile manipulators and various interaction scenarios, including sparring, handshaking, rock-paper-scissors, or dancing. Our results demonstrate a promising path toward coordinated behaviors across morphologically distinct characters via cross-embodiment interaction learning. Mastering these behaviors is essential for robots, particularly non-humanoid ones, to function seamlessly in human environments. Robots may conduct daily activities such as object handover or participate in social rituals. Such interactions are critical not only for effective collaboration but also for fostering trust, acceptance, and intuitive communication between humans and machines. Beyond robotics, the ability to synthesize realistic physical interactions is also central to computer graphics and animation, where the goal is to produce lifelike and engaging character behaviors in films, games, and virtual worlds. Mastering physical interaction skills presents significant challenges due to the complex and dynamic nature of inter-agent coordination.