TimewarpVAE: Simultaneous Time-Warping and Representation Learning of Trajectories

Continuous trajectories are inherently infinite-dimensional objects that can vary in complex ways in both time and space. However, in many practical situations, they contain intrinsic sources of variability that can be wellapproximated by projection onto a low-dimensional manifold. For instance, when a human demonstrates trajectories for a robot, it is useful for the robot to learn to model the most expressive latent factors controlling the spatial paths of the demonstration trajectories. For certain types of demonstrations, such as in gesture control or quasistatic manipulation, it is highly advantageous to explicitly separate the exact timing of the trajectory from the spatial latent factors. As an illustrative example, consider trying to average two samples from a handwriting dataset generated by humans drawing the letter "A" in the air (Chen et al., 2012). If we scale two trajectories linearly in time so that both their timestamps go from 0 to 1, and then average the two trajectories at each timestep, the resulting average does not maintain the style of the "A"s. This is because the average is taken between parts of the two trajectories that do not naturally correspond to each other. An example of this averaging, with lines showing examples of points that are averaged, is shown in Figure 1a. A common approach like Dynamic Time Warping (DTW) (Sakoe & Chiba, 1978) can lead to unintuitive results.