TrajDiff: End-to-end Autonomous Driving without Perception Annotation