Pixel is a Barrier: Diffusion Models Are More Adversarially Robust Than We Think