A Hierarchical Hybrid Learning Framework for Multi-agent Trajectory Prediction