Artificial Intelligence and Legal Analysis: Implications for Legal Education and the Profession

LLMs were tested on legal reasoning tasks involving rule analysis and analogical reasoning. The results show that LLMs can conduct basic IRAC analysis, but are limited by brief responses lacking detail, an inability to commit to answers, false confidence, and hallucinations. The study compares legal and non-legal LLMs, identifies shortcomings, and explores traits that may hinder their ability to "think like a lawyer."