Alpha Excel Benchmark

ABSTRACT This study presents a novel benchmark for evaluating Large Language Models (LLMs) using challenges derived from the Financial Modeling W orld Cup (FMWC) Excel competitions. W e introduce a methodology for converting 113 existing FMWC challenges into programm atically evaluable JSON formats and use this dataset to compare the performance of several leading LLMs. Our findings demonstrate significant variations in performance across different challenge categories, with models showing specific strengths in pattern recognition tasks but struggling with complex numerical reasoning. The benchmark provides a standardized framework for assessing LLM capabilities in realistic business - oriented tasks rather than abstract academic problems. This resear ch contributes to the growing field of AI benchmarking by establishing proficiency among the 1.5 billion people who daily use Mic rosoft Exc el as a meaningful evaluation metric that bridges the gap between academic AI benchmarks and practical business applications. INTRODUCTION The recent rapid advancement of Large Language Models (LLMs) has sparked interest in developing specialized benchmarks to evaluate their capabilities across various domains. While existing benchmarks often focus on natural language understanding, programmi ng, or reasoning abilities in abstract contexts, there remains a notable gap in benchmarks that assess performance on practical business tasks (Brown et al., 2020). Microsoft Excel, being one of the most widely used business software tools globally, presen ts an opportunity to create tasks that simultaneously test multiple dimensions of LLM capabilities, including numerical reasoning, pattern recognition, rule comprehension, file conversion, and problem - solving strategies. The Financial Modeling W orld Cup (FMWC), established in 2020, has emerged as a premier global competition testing advanced Excel skills through creative challenges that range from financial modeling to game simulations implemented in spreadsheets (Grigolyu novich, 2022).