LLMs Understand Glass-Box Models, Discover Surprises, and Suggest Repairs

Large language models (LLMs) offer the potential to automate data science through natural language interfaces, but it is difficult to embed complex models or datasets in confined context windows. While GPT-4 has a context window size of up to 32k tokens, paying equal attention to all parts of the context remains a challenge [1] and the practicality of lengthy context windows is questionable. Machine learning models often involve billions of parameters, accentuating the need for compact, modular function representations that more easily interface with LLMs. In this paper, we show that LLMs pair remarkably well with interpretable models that are decomposable into modular components. Specifically, we show that GPT-4 is able to describe, interpret and debug univariate graphs, and by applying a form of chain-of-thought reasoning[2], GPT-4 can understand Generalized Additive Models (GAMs). GAMs [3, 4] represent complex outcomes as sums of univariate component functions (graphs); thus, by analyzing each of these component functions in turn, the LLM does not need to understand the entire model at once. After analyzing and summarizing each graph, the LLM can operate on component summaries to produce model-level analyses. This modularity simplifies the application of LLMs to data science and machine learning and enables LLM-based analyses to scale to very large datasets while staying within small context windows.