ReGenNet: Towards Human Action-Reaction Synthesis

In this paper, we focus on generative models for static scenes and objects, while the dynamic human actionreaction human action-reaction synthesis, i.e., generating human reactions synthesis for ubiquitous causal human-human interactions given the action sequence of another as conditions. is less explored. Human-human interactions We believe this task will contribute to many applications in can be regarded as asymmetric with actors and reactors AR/VR, games, human-robot interaction, and embodied AI. in atomic interaction periods. In this paper, we comprehensively Modeling human-human interactions is a challenging analyze the asymmetric, dynamic, synchronous, task with the following features: 1) Asymmetric, i.e., the and detailed nature of human-human interactions and propose actor and reactor play asymmetric roles during a causal interaction, the first multi-setting human action-reaction synthesis where one person acts, and the other reacts [78]; benchmark to generate human reactions conditioned on 2) Dynamic, i.e., during the interaction period, the two people given human actions. To begin with, we propose to annotate constantly wave their body parts, move close/away, and the actor-reactor order of the interaction sequences change relative orientations, spatially and temporally; 3) for the NTU120, InterHuman, and Chi3D datasets. Based Synchronous, i.e., typically, one person responds instantly on them, a diffusion-based generative model with a Transformer with others such as an immediate evasion when someone decoder architecture called ReGenNet together with throws a punch, thus the online generation is required; 4) an explicit distance-based interaction loss is proposed to Detailed, i.e., the interaction between humans involves not predict human reactions in an online manner, where the future only coarse body movements together with relative position states of actors are unavailable to reactors.