Rethinking the Outlier Distribution in Large Language Models: An In-depth Study