Self-diffusion for Solving Inverse Problems