VidMan: Exploiting Implicit Dynamics from Video Diffusion Model for Effective Robot Manipulation