Less is More: Efficient Brain-Inspired Learning for Autonomous Driving Trajectory Prediction