Structured Graph Network for Constrained Robot Crowd Navigation with Low Fidelity Simulation