Prompt Diffusion Robustifies Any-Modality Prompt Learning