Mind your Language (Model): Fact-Checking LLMs and their Role in NLP Research and Practice

LLM-based systems are increasingly deployed in user-facing products in contexts ranging from health (Graber-Stiehl 2023; Harrer 2023) to education (Kasneci et al. 2023), which translates into real-life impacts on thousands of people. Yet despite the many research articles on LLMs, their very definition remains unclear, and much empirical work is based on ideas that lack a solid theoretical grounding or rigorous empirical evidence. To contribute to more rigorous scientific discussion of LLMs, we propose a more precise definition of what factors are necessary and sufficient for an NLP model to be considered an LLM ( 2), and we examine several common claims about LLM properties and functionality ( 3). We then consider the impact that these claims and assumptions have on NLP research ( 4), and we conclude with proposals for maintaining rigor and diversity in NLP research and practice ( 5). Our work draws heavily on both empirical studies and socio-technical critiques of LLMs. 2. What Counts as an LLM? The term "large language model" is used both within the NLP community (e.g. in the call for papers of the Theme Track of EMNLP 2023) as well as in news articles and in U.S. Senate hearings. Despite its ubiquity, its definition is far from clear. For instance, how many parameters should a neural network have to qualify as an LLM? And do only Transformer-type architectures qualify?