How to Use Diffusion Priors under Sparse Views?