Scaling Laws for Economic Productivity: Experimental Evidence in LLM-Assisted Translation

The amount of training compute used by frontier large language models (LLMs) increased by 5000x between the release of GPT-2 in 2019 and GPT-4 in 2023 and estimates from Epoch AI suggest a similar increase over the next six years. How does this massive increase in model training compute map onto performance? The empirical machine learning literature has derived remarkably consistent'scaling laws' suggesting a strong relationship between a model's training compute and model perplexity, a measure of model loss, across more than seven orders of magnitude. But there is so far a very limited understanding of how this reduction in perplexity affects key economic and social outcomes. This paper aims to offer the first experimental evidence on this question by conducting a randomized controlled trial(RCT) involving 300 professional translators conducting 1800 tasks of varying complexities. The participants were randomly assigned to either treatment groups where they could utilize one of thirteen LLMs of differing model training compute to help them complete their task or to a control group where they completed tasks without any AI assistance. Participants face high-powered incentives with significant bonus payments for high-quality tasks as evaluated by three experienced professionals in the field. The key outcome variables, therefore, were how translator's time taken, quality of tasks completed, and earnings per minute (inclusive of bonuses) varied by model training compute.