Demystifying Singular Defects in Large Language Models