DiWA: Diffusion Policy Adaptation with World Models