Explaining Context Length Scaling and Bounds for Language Models

A wide variety of work is proposed to discuss the impact of context length: some shows long irrelevant context Long Context Language Models have drawn would worsen performance for LMs(Xu et al., 2024; great attention in the past few years. There has Levy et al., 2024); some shows long context would improve been work discussing the impact of long context performance in a way summarized as Scaling Laws(Xiong on Language Model performance: some find that et al., 2024); while work in other domains like time series long irrelevant context could harm performance, shows long relevant context would hurt performance while some experimentally summarize loss reduction (Shi et al., 2024). This calls for a more thorough understanding by relevant long context as Scaling Laws. of how context length affects Language Models' This calls for a more thorough understanding on performance.. how long context impact Language Modeling. In this work, we (1) propose a clean and effective Previously, theories have been proposed to explain the Scaling theoretical framework on explaining the impact Laws with respect to the data set and the size of the of context length to Language Modeling, from an model(Bahri et al., 2024; Sharma & Kaplan, 2020). However, Intrinsic Space perspective; and (2) conduct experiments these theories do not study how context length impact on natural language and synthetic data, Language Modeling, thus they cannot contribute directly to validating our proposed theoretical assumptions the problem.