TuRTLe: A Unified Evaluation of LLMs for RTL Generation

--Rapid advancements in LLMs have driven the adoption of generative AI in domains like Electronic Design Automation (EDA). Within the field of software development, EDA presents unique challenges derived from specific requirements of generated RTL code; RTL code must not only be syntactically correct and functionally accurate, but also synthesizable by hardware generators, while matching performance, power and area (PPA) constraints. These additional requirements introduce complexities that existing code-generation benchmarks often fail to capture, limiting their effectiveness in evaluating LLMs for RTL generation. Using this framework, a diverse set of forty open LLMs are assesed, tracking their strengths and weaknesses in EDA-specific tasks. Our results identify the best match for specific tasks ( e.g., base models are better in module completion tasks, instruct-tuned models are better in specification-to-RTL tasks), while finding that recent models with autoregressive reasoning chain perform the best overall. Some benchmarks, particularly within syntax correctness, show signs of saturation, while others remain as open problems for LLMs. Advancements in large language models (LLMs) have unlocked new possibilities across a wide range of domains [1], [2]. Domain-specific LLMs have gained significant attention due to their strong performance in specialized tasks, including financial engineering [3], biomedical research [4], and scientific computing [5], [6]. In software related tasks, LLM can suggest code snippets, solve common coding challenges, and provide explanations of complex concepts [7]-[10]. In the field of Electronic Design Automation (EDA), researchers are increasingly exploring the use of LLMs to accelerate hardware design [11]-[13].