The Achilles' Heel of LLMs: How Altering a Handful of Neurons Can Cripple Language Abilities