LegalBench: Prototyping a Collaborative Benchmark for Legal Reasoning

Advances in language modeling are changing how American lawyers and administrators envision the practice of law [13]. In transactional settings, computational language tools are already being used in document review [18], and have illustrated promise for more sophisticated tasks like due diligence [4]. In administrative and civil settings [12, 14], many have identified the potential for computational tools to improve the accessibility of legal services [10, 26, 27, 30], thereby alleviating the United States' long standing access-to-justice crisis [8]. Unsurprisingly, the high risk nature of these tools--and their position in society--has inspired calls for better, law specific evaluation and auditing regimes [11]. The potential for impactful computational legal language tools has been magnified by the development of language Foundation Models (FM)--large scale models trained on massive corpora of text [2].