EHRNoteQA: An LLM Benchmark for Real-World Clinical Practice Using Discharge Summaries

EHRNoteQA in two formats: open-ended and multi-choice question answering, and propose a reliable evaluation method for each. We evaluate 27 LLMs using EHRNoteQA and examine various factors affecting the model performance ( e.g., the length and number of discharge summaries).