Cherry on Top: Parameter Heterogeneity and Quantization in Large Language Models

This paper reveals the phenomenon of parameter heterogeneity in large language models (LLMs). We find that a small subset of cherry'' parameters exhibit a disproportionately large influence on model performance, while the vast majority of parameters have minimal impact. This heterogeneity is found to be prevalent across different model families, scales, and types. Motivated by this observation, we propose CherryQ, a novel quantization method that unifies the optimization of mixed-precision parameters. CherryQ identifies and preserves the critical cherry parameters in high precision while aggressively quantizing the remaining parameters to low precision.