A Law of Next-Token Prediction in Large Language Models

Large language models (LLMs) have been widely employed across various application domains, yet their black-box nature poses significant challenges to understanding how these models process input data internally to make predictions. In this paper, we introduce a precise and quantitative law that governs the learning of contextualized token embeddings through intermediate layers in pre-trained LLMs for next-token prediction. Our findings reveal that each layer contributes equally to enhancing prediction accuracy, from the lowest to the highest layer--a universal phenomenon observed across a diverse array of open-source LLMs, built on architectures such as Transformer, RWKV, and Mamba. We demonstrate that this law offers new perspectives and insights to inform and guide practices in LLM development and applications, including model scaling, pre-training tasks, and information flow. Overall, our law enables more fine-grained approaches to the design, training, and interpretation of LLMs through scrutinizing their internal data processing mechanisms. The rapid advancement of large language models (LLMs) has profoundly impacted various fields, including mathematical discovery [34], medical diagnosis [5], genomic research [19, 15], and education [42]. Despite their transformative and widespread adoption, the deployment of LLMs is often impeded by a lack of understanding of how these enormous, complex black-box models internally process data to generate predictions [32]. Without understanding the prediction mechanisms, practitioners face challenges in interpreting these predictions for decision-making.