A Simple Model of Inference Scaling Laws

Advancements in deep learning have demonstrated that the performance of neural networks scales predictably as a function of model size, data size, and computational resources [Hestness et al., 2017, Kaplan et al., 2020a, Rosenfeld et al., 2020, Henighan et al., 2020a]. These trends, known as neural scaling laws, have motivated research into understanding how scaling influences model performance in a range of domains, in particular, Large Language Models (LLMs) [Brown et al., 2020, Hoffmann et al., 2022]. However, scaling during inference--the process by which a trained model makes predictions on new data--has received less attention. Recent works have shown empirically that LLMs can gain substantial benefits from repeated prompts to perform better on difficult tasks such as coding and formal proofs, where verification of the correct answer can be done [Brown et al., 2024, Snell et al., 2024, Bansal et al., 2024]. These works demonstrate that the performance of weaker models can be amplified without further training, by simply repeating inference trials. A natural question then arises: Can we interpret, or predict the inference scaling behavior of a model with repeated attempts? To answer this question, we propose a simple toy model that isolates the inference scaling laws which dictate how certain performance metrics improve as a function of the number of inference attempts. Inspired by the work of Hutter [2021], which introduced a model to study scaling behavior for memorization and generalization, we devise a simple setting to capture the effect of repeated inference attempts, focusing on the coverage metric, also known as pass@k. In this work, we present analytical predictions for coverage from a probabilistic perspective and demonstrate how inference improves with the number of repeated trials in a predictable way, which matches the observed behavior in Brown et al. [2024] and Snell et al. [2024].