Collision Avoidance Detour for Multi-Agent Trajectory Forecasting