Comparison of Unsupervised Metrics for Evaluating Judicial Decision Extraction

The integration of artificial intelligence (AI) into the legal domain has revolutionized judicial processes, enabling tasks such as legal judgment prediction (LJP), case summarization, precedent retrieval, and automated legal research. Text extraction, the process of isolating seven semantically meaningful segments--referred to as blocks--from unstructured judicial decisions, is a cornerstone of these applications. These blocks include plaintiff demands, plaintiff arguments, defendant arguments, court evaluation of evidence, judicial reasoning steps, applicable legal norms, and court decision. Accurate extraction is critical, as errors can lead to misinterpretations of case facts, biased predictions, or inefficiencies in judicial workflows, potentially undermining justice delivery in high-stakes contexts. Evaluation metrics are essential for quantifying extraction quality, enabling iterative model improvements and ensuring reliability. Traditional metrics rely on annotated ground truth, which is resource-intensive to produce, particularly for legal texts characterized by verbose narratives, domain-specific terminology, and jurisdiction-specific nuances. The scarcity of annotated legal corpora has driven the development of unsupervised metrics that leverage intrinsic document properties, such as term frequencies, semantic coherence, and structural patterns. These metrics must capture surface-level accuracy, semantic fidelity, logical structure, and legal-specific elements like citations and temporal consistency, while addressing ethical concerns such as fairness and neutrality in AI-driven legal systems [1, 2].