Stochastic Trajectory Prediction under Unstructured Constraints