Predicting LLM Reasoning Performance with Small Proxy Model

Given the prohibitive cost of pre-training large language models, it is essential to leverage smaller proxy models to optimize datasets before scaling up. However, this approach becomes challenging for reasoning capabilities, which exhibit emergent behavior that only appear reliably at larger model sizes, often exceeding 7B parameters. In response, leveraging smaller models as a proxy for larger model performance has been a key direction through the establishment of empirical scaling laws for prediction (Kaplan et al., 2020; Hoffmann et al., 2022b) or derivation of pre-training dataset rank invariance across scale (Magnusson et al., 2025). 's reasoning trace (Wei et al., 2022b) as the gold label Y Each token i's NLL is weighted by the frontier model's confidence in that token (MinMax normalized). To handle tokenizer mismatches between proxy and frontier models, we compute weights at the letter level and average within tokens. 1 Figure 2: Using MA TH500 as an example benchmark, given the same data source OLMo-Mix-1124 (OLMo et al., 2024), smaller models exhibit more noise and get the direction wrong, making it challenging to use smaller models to proxy larger model performance. Extended visualization across other reasoning benchmarks are available in Appendix A. However, the literature on the emergence of reasoning performance as we scale model size (Wei et al., 2022a; Almazrouei et al., 2023; Du et al., 2024) suggest that there may be a limit to how small theses proxy models can be. They demonstrate that reasoning capabilities only appear when models are sufficiently large in size.