AdaWM: Adaptive World Model based Planning for Autonomous Driving