Reinforcement learning has shown great promise for synthesizing realistic human behaviors by learning humanoid control policies from motion capture data. However, it is still very challenging to reproduce sophisticated human skills like ballet dance, or to stably imitate long-term human behaviors with complex transitions. The main difficulty lies in the dynamics mismatch between the humanoid model and real humans. That is, motions of real humans may not be physically possible for the humanoid model. To overcome the dynamics mismatch, we propose a novel approach, residual force control (RFC), that augments a humanoid control policy by adding external residual forces into the action space.

Weaknesses: The biggest weakness of the paper is that it seems to be making claims that are not backed up by the results. For example, the paper states that "show for the first time humanoid control policies thatare capable of highly agile ballet dancing moves" and basically argues that policies of similar complexity have not been learned before. However, the DeepMimic paper showed really complex motor skills including back-flips, a pirouette-like spin, Karate moves and other motions that are at least on a similar level of complexity. Further, the paper mostly focuses on tracking the observed motions (potentially small variations thereof). But in imitation learning it is very important to show the generalization capabilities of a learned policy; otherwise a trajectory would suffice.

The reviewers highly appreciated the rebuttal which successfully addressed many doubts. While the general idea of adding external forces to aid the learning process has been previously in other settings, the application to imitation learning for humanoid agents is certainly innovative, the results are very impressive, and the rebuttal convincingly argues for the potential of application beyond simulations.

Reinforcement learning has shown great promise for synthesizing realistic human behaviors by learning humanoid control policies from motion capture data. However, it is still very challenging to reproduce sophisticated human skills like ballet dance, or to stably imitate long-term human behaviors with complex transitions. The main difficulty lies in the dynamics mismatch between the humanoid model and real humans. That is, motions of real humans may not be physically possible for the humanoid model. To overcome the dynamics mismatch, we propose a novel approach, residual force control (RFC), that augments a humanoid control policy by adding external residual forces into the action space.