This article is part of Demystifying AI, a series of posts that (try to) disambiguate the jargon and myths surrounding AI. ChatGPT and other large language models (LLM) have proven to be useful for tasks other than generating text. However, in some fields, their performance is confusing. One such area is math, where LLMs can sometimes provide correct solutions to difficult problems while at the same time failing at trivial ones. There is a body of research that explores the capabilities and limits of LLMs in mathematics.

This article is part of our coverage of the latest in AI research. This week, OpenAI released ChatGPT, another fascinating large language model (LLM) based on its flagship GPT series. ChatGPT, which is available as a free demo at the time of this writing, is a model that has been specialized for conversational interactions. As with most things regarding LLMs, the release of ChatGPT was followed by controversy. Within hours, the new language model became a Twitter sensation, with users posting screenshots of ChatGPT's impressive achievements and disastrous failures. However, when looked at from the broad perspective of large language models, ChatGPT is a reflection of the short but rich history of the field, representing how far we have come in just a few years and what fundamental problems remain to be solved.

Large language models (LLMs), like OpenAI ChatGPT and Google LaMDA, are impressive, being competent in many aspects. At the same time, LLMs are incompetent in many ways. LLMs are evolving, and new players are joining. What further progress may be possible? Moreover, we may ask a question relevant to almost all players in the world of LLMs, from students, researchers, engineers, entrepreneurs, venture capitalists, officers, to the public crowd: Where is the boundary for large language models?

Are Large Language Models the path to AGI? The development of large language models has been a major milestone in narrow AI, machine learning and deep learning in recent years. There are a number of LLMs, as pictured below, and the most prominent include GPT-3, ChatGPT (OpenAI), BERT, T5 (Google) or Wu Dao (Beijing Academy of Artificial Intelligence), MT-NLG (Microsoft), META/Galactica. Such Large Language Models (LLMs) can give us the impression of being capable of doing a lot of things. Their training from huge web corpora in an unsupervised way with the latest neural network architectures, Transformer Models, consisting of a Deep Learning Model and equipped with an Attention Mechanism.

Artificial-intelligence systems that can churn out fluent text, such as OpenAI's ChatGPT, are the newest darlings of the technology industry. But when faced with mathematical queries that require reasoning to answer, these large language models (LLMs) often stumble. A line parallel to y 4x 6 passes through (5, 10). What is the y-coordinate of the point where this line crosses the y-axis? Although LLMs can sometimes answer these types of question correctly, they more often get them wrong. In one early test of its reasoning abilities, ChatGPT scored just 26% when faced with a sample of questions from the'MATH' data set of secondary-school-level mathematical problems1. This is to be expected: given input text, an LLM simply generates new text in accordance with statistical regularities in the words, symbols and sentences that make up the model's training data.