Weak Diffusion Priors Can Still Achieve Strong Inverse-Problem Performance