Large Language Models and Emergence: A Complex Systems Perspective

Large Language Models (LLMs) are deep neural networks that, through training on huge amounts of text, learn to accurately predict the next word (or token) in a text. It has been surprising to many that next-token prediction has lead to impressive abilities, such as learning of syntax, code generation, writing in any style, and factual recall. It has been claimed in the LLM literature that, as the number of network parameters and amount of training data is scaled up, certain capabilities arise suddenly and unexpectedly, a phenomenon that these writers term "emergence". For example, Wei et al. [1] write, "we define emergent abilities of large language models as abilities that are not present in smaller-scale models but are present in large-scale models; thus they cannot be predicted by simply extrapolating the performance improvements on smaller-scale models." And in a recent review of emergent abilities in LLMs Berti et al. [2] survey around 100 papers the majority of which equate emergence with the discontinuous appearance of abilities with increasing data or model size.