Learning robot control policies by imitation [31]isan appealing approach toskill acquisition and has been successfully applied to several tasks, including locomotion, grasping, and even table tennis [8, 2, 25]. In this paradigm, expert demonstrations of robot motion are first recorded via kinesthetic teaching, teleoperation, or other input modalities.

In turn, this embedding isthen used to generate the hyper-parameters of the low-level controller (line 152), namely28 weights,currentphase,anddesiredphaseprogression (line173). As the reviewer pointed out, multi-staging requires a38 significant amount ofadditional (human) feature engineering, which would, atthesame time, limit ourapproach to39 these features and may also be fragile in terms of generalizing to new words. Our framework learns how language40 affects the behavior (type, goal position, velocity,etc.)

We thank all reviewers for their constructive feedback! We collected 200 such descriptions (40 per annotator). New Users", users typed an instruction and saw the result in a physics-based simulation in real-time (line 263). R2 Faster R-CNN isn't trained on data that looks anything like this. FPFH would not be applicable since it is a 3D point-cloud approach requiring access to a depth camera.