Legal Compliance Evaluation of Smart Contracts Generated By Large Language Models

--Smart contracts can implement and automate parts of legal contracts, but ensuring their legal compliance remains challenging. Existing approaches such as formal specification, verification, and model-based development require expertise in both legal and software development domains, as well as extensive manual effort. Given the recent advances of Large Language Models (LLMs) in code generation, we investigate their ability to generate legally compliant smart contracts directly from natural language legal contracts, addressing these challenges. We propose a novel suite of metrics to quantify legal compliance based on modeling both legal and smart contracts as processes and comparing their behaviors. We select four LLMs, generate 20 smart contracts based on five legal contracts, and analyze their legal compliance. We find that while all LLMs generate syntactically correct code, there is significant variance in their legal compliance with larger models generally showing higher levels of compliance. We also evaluate the proposed metrics against properties of software metrics, showing they provide fine-grained distinctions, enable nuanced comparisons, and are applicable across domains for code from any source, LLM or developer . Our results suggest that LLMs can assist in generating starter code for legally compliant smart contracts with strict reviews, and the proposed metrics provide a foundation for automated and self-improving development workflows. Blockchains are increasingly used for multi-party business processes [1], with users making direct agreements via immutable programs known as smart contracts [2]. With the tokenized asset market projected to reach US $16.1 trillion by 2030 [3], businesses are rapidly developing smart contract-based services on blockchain platforms. To avoid significant legal, financial, and reputational risks, these smart contracts must comply with legal constraints that arise from legal contracts between stakeholders [4], [5] and regulatory frameworks in the operating jurisdiction [6]. Manually implementing legally compliant code is a time-consuming and error-prone process, especially for complex multi-party agreements. With recent advancements in large language models (LLMs) such as GPT -4, there has been a growing interest in using generative models to write code.