AbsenceBench: Language Models Can't Tell What's Missing Harvey Yiyun Fu,1, Aryan Shrivastava1, Jared Moore2 Peter West2, Chenhao Tan1, Ari Holtzman1 1University of Chicago 2Stanford University

Large language models (LLMs) are increasingly capable of processing long inputs and locating specific information within them, as evidenced by their performance on the Needle in a Haystack (NIAH) test. However, while models excel at recalling surprising information, they still struggle to identify clearly omitted information. We introduce AbsenceBench to assesses LLMs' capacity to detect missing information across three domains: numerical sequences, poetry, and GitHub pull requests. AbsenceBenchasks models to identify which pieces of a document were deliberately removed, given access to both the original and edited contexts. Despite the apparent straightforwardness of these tasks, our experiments reveal that even state-of-the-art models like Claude-3.7-Sonnet