Less is More: Lean yet Powerful Vision-Language Model for Autonomous Driving