A Large-Language Model Framework for Relative Timeline Extraction from PubMed Case Reports

Jing Wang, Ph.D.1, Jeremy C. Weiss, M.D., Ph.D.1 1National Library of Medicine, Bethesda, Maryland, USA Abstract Timing of clinical events is central to characterization of patient trajectories, enabling analyses such as process tracing, forecasting, and causal reasoning. However, structured electronic health records capture few data elements critical to these tasks, while clinical reports lack temporal localization of events in structured form. We present a system that transforms case reports into textual time series--structured pairs of textual events and timestamps. We contrast manual and large language model (LLM) annotations (n=320 and n=390 respectively) of ten randomly-sampled PubMed open-access (PMOA) case reports (N=152,974) and assess inter-LLM agreement (n=3,103 N=93). We find that the LLM models have moderate event recall (O1-preview: 0.80) but high temporal concordance among identified events (O1-preview: 0.95). Introduction Clinical event timelines are key analytic devices in use in areas ranging from the visualization of patient trajectories in electronic health records to the development and update of clinical practice guidelines. While many automated documentation systems capture structured health information in relational databases with timestamping, textual data modalities often lack temporal granularity beyond creation and submission dates, despite being the main tool care providers use to document and communicate patient history and care planning.