LongGenBench: Benchmarking Long-Form Generation in Long Context LLMs

In evaluating the long-context capabilities of large language models (LLMs), benchmarks such as "Needle-in-a-Haystack" (NIAH), Ruler, and Needlebench are commonly used. While these benchmarks measure how well models understand longcontext input sequences, they do not effectively gauge the quality of long-form text generation--a critical aspect for applications such as design proposals and creative writing. To address this gap, we have introduced a new long-form text evaluation benchmark, LongGenBench, which tests models' ability to identify specific events within generated long text sequences. In this benchmark, we prompt long-context LMs to create long-form text that must include particular events or constraints and evaluate their ability to incorporate these elements. We evaluated ten longcontext LMs across four distinct scenarios, three types of prompt instructions, and two different generation-length settings (16K and 32K). Although these models perform well on NIAH benchmarks, none demonstrated satisfactory performance on the LongGenBench, raising concerns about their ability to generate coherent long-form text that follows instructions. Recent advances in large language models (LLMs) have significantly enhanced their ability to process and generate long text sequences, which is essential for various natural language processing tasks (Xiong et al., 2023; Bai et al., 2023a). For instance, GPT-4 (Achiam et al., 2023) and LLaMa-3.1 (Dubey et al., 2024) manage context windows of up to 128K tokens; Claude 2.1 (Anthropic, 2024a) supports up to 200K tokens; and the Claude 3 series (Anthropic, 2024b) handles inputs exceeding 1 million tokens.