The current dominant paradigm in sensorimotor control, whether imitation or reinforcement learning, is to train policies directly in raw action spaces such as torque, joint angle, or end-effector position. This forces the agent to make decision at each point in training, and hence, limits the scalability to continuous, high-dimensional, and long-horizon tasks. In contrast, research in classical robotics has, for a long time, exploited dynamical systems as a policy representation to learn robot behaviors via demonstrations. These techniques, however, lack the flexibility and generalizability provided by deep learning or deep reinforcement learning and have remained under-explored in such settings. In this work, we begin to close this gap and embed dynamics structure into deep neural network-based policies by reparameterizing action spaces with differential equations. We propose Neural Dynamic Policies (NPDs) that make predictions in trajectory distribution space as opposed to prior policy learning methods where action represents the raw control space. The embedded structure allows us to perform end-to-end policy learning under both reinforcement and imitation learning setups. We show that NDPs achieve better or comparable performance to state-of-the-art approaches on many robotic control tasks using both reward-based training and demonstrations.

Weaknesses: The biggest limitation is the lack of imitation learning experiments. The authors chose to conduct an imitation learning experiment in a digit writing domain. However, the authors ran extensive RL experiments in a variety of robotic manipulation domains - I strongly advise using a policy trained on these domains as an expert (preferably not trained with NDP) and running behavioral cloning experiments using NDP and comparing against other action spaces and policy architectures. This would help decouple the benefit of NDP for exploration from the benefit of NDP as an action representation for control and modeling action sequences and should be a fairly straightforward experiment to run. It would also be interesting to see the effect of the control frequency and subsampling expert action sequences in the data - something that NDP is uniquely suited to do.

The paper proposes a very interesting, novel policy representation with extensive evaluations both for imitation learning and reinforcement learning. The reviewers highly appreciated the additional insights and experiments in the rebuttal.

The current dominant paradigm in sensorimotor control, whether imitation or reinforcement learning, is to train policies directly in raw action spaces such as torque, joint angle, or end-effector position. This forces the agent to make decision at each point in training, and hence, limits the scalability to continuous, high-dimensional, and long-horizon tasks. In contrast, research in classical robotics has, for a long time, exploited dynamical systems as a policy representation to learn robot behaviors via demonstrations. These techniques, however, lack the flexibility and generalizability provided by deep learning or deep reinforcement learning and have remained under-explored in such settings. In this work, we begin to close this gap and embed dynamics structure into deep neural network-based policies by reparameterizing action spaces with differential equations.