Data movement limits to frontier model training

We present a theoretical model of distributed training, and use it to analyze how far dense and sparse training runs can be scaled. FLOP, two orders of magnitude above the largest training run to date, suggesting the arrival of fundamental barriers to scaling in three years given recent rates of growth. FLOP is infeasible even at low utilization. However, more aggressive batch size scaling and/or shorter and fatter model shapes, if achievable, have the potential to permit much larger training runs. An interactive version of our model will shortly be accessible here. In this work, we address unexamined fundamental questions about limits to scaling in the future: Q1 Given present-day algorithms, GPUs, and interconnects, what is the biggest training run that can be performed within a fixed duration, before intra-and inter-GPU data movement starts to seriously worsen utilization or even render it impossible? Q2 How far might this limit be extended, and what algorithmic or hardware progress can achieve that? Answering these questions empirically would require millions of GPUs and large-scale engineering efforts, so we instead approach them theoretically. In doing so, we develop a simulator that can find optimal training run configurations accounting for the factors that we identify as fundamental. We focus on GPUs, but our theoretical model and findings are broadly applicable to other accelerators, and even groups of accelerators. A2 Improved hardware interconnects may buy no more than two orders of magnitude in training run size, assuming technology anything like the current paradigm. Beyond that, the critical innovations must come from machine learning algorithms: The key challenge is transforming two serial dependencies -- between batches and between layers -- into opportunities for parallelism, by making batch sizes bigger (perhaps enabled by sparsity) and models wider and shallower. Achieving these goals may be quite difficult in practice. However, with innovations in scaling (such as techniques to enable much larger batch sizes) or dramatic increases in network bandwidth coupled with a 10 reduction in interand intra-GPU latency, training runs can be at least a few orders of magnitude larger (right).