SortBench: Benchmarking LLMs based on their ability to sort lists

SortBench: Benchmarking LLMs based on their ability to sort lists Steffen Herbold steffen.herbold@uni-passau.de Faculty of Computer Science and Mathematics University of Passau Passau, Germany Abstract Sorting is a tedious but simple task for human intelligence and can be solved fairly easily algorithmically. However, for Large Language Models (LLMs) this task is surprisingly hard, as some properties of sorting are among known weaknesses of LLMs: being faithful to the input data, logical comparisons between values, and strictly differentiating between syntax (used for sorting) and semantics (typically learned by embeddings). Within this paper, we describe the new SortBench benchmark for LLMs that comes with different difficulties and that can be easily scaled in terms of difficulty. We apply this benchmark to seven state-of-the-art LLMs, including current test-time reasoning models. Our results show that while the o3-mini model is very capable at sorting in general, even this can be fooled if strings are defined to mix syntactical and semantical aspects, e.g., by asking to sort numbers written-out as word. Furthermore, all models have problems with the faithfulness to the input of long lists, i.e., they drop items and add new ones. Our results also show that test-time reasoning has a tendency to overthink problems which leads to performance degradation. Finally, models without test-time reasoning like GPT-4o are not much worse than reasoning models. 1 Introduction Sorting is a basic skill that humans acquire at an early age already: we learn the order of numbers, the alphabet, and we also learn to sort by shapes or similar aspects. While sorting with very large numbers of items is a tedious activity for humans, we can solve it reliably nonetheless. Moreover, algorithmic sorting strategies are well-known and range from intuitive (bubble sort, insertion sort) to more optimized variants (quicksort, mergesort). Thus, from both a human intelligence perspective as well as from the algorithmic complexity required to solve the problem, sorting is a simple problem. Modern Large Language Models (LLMs) based on decoder-only transformers (Radford et al., 2018), instruction fine-tuning (Zhang et al., 2023), and even advanced chain-of-thought reasoning (Jaech et al., 2024) are showing increasingly strong capabilities at even solving hard problems like advanced math (AoPS Online, 2024) or the ARC challenge (Price, 2024).