Degrees of Freedom Matter: Inferring Dynamics from Point Trajectories