Appendix 1

Pi,jCi,j γH(P) subjectto P Rt t+,PT1t =1t,P1t =1t, (6) where Pi,j is the transport plan andCi,j is the ground metric that measures the distance between point i in the source andj in the target. This will induce some smoothness and wiggle room in the solutionofourobjective. To increase the diversity of the observed trajectories, we inject Gaussian noise (σ = 0.05) into trajectories by perturbing the initial velocities. Since two-body systems are non-chaotic systems, we divide training set and testing set such that for training set[mmin,mmax] = [0.8,1.2], while testing set[mmin,mmax] = [0.9,1.3] to create domain distribution shifting. The initial velocities of all bodies are based on their initial positions by rotating itby 90 andscalingitbyr1.5.