What Makes Quantization for Large Language Models Hard? An Empirical Study from the Lens of Perturbation