What Truly Matters in Trajectory Prediction for Autonomous Driving?