Supplementary Materials for M

A.1 Extraction Process As mentioned in Section 2, the patient notes used for M The motivation of this track was to challenge participants to obtain relevant articles that can help answer potential questions for a particular patient note. The patient notes 2014 and 2015 are synthetic patient notes hand-written by individuals with medical training, but the 2016 dataset consists of real patient summaries coming from electronic health records. TREC Clinical Trials 125 137.7 This track consists of 125 patient notes, where 50 notes are from the year of 2021 and 75 notes are from the year of 2022. This track was meant to have participants retrieve previous clinical trials from ClinicalTrials.gov that best match the symptoms described in the patient note. The notes from both tracks are synthetic notes written by individuals with medical training meant to simulate an admission statement from an electronic health record (EHR). MedQA-USMLE 12,893 135.8 Questions from a multiple-choice form a professional medical board exam which can include patient summaries and will ask questions about particular issues based on the summary. We used GPT-3.5-turbo to identify eligible patient notes for each calculator. We did this by shortlisting the notes that had at least one relevant required for a given calculator. We kept the notes that had all of the numeric parameters needed for each calculator. These notes were also filtered to have enough categorical variables inferred such that 50% of the total number of attributes were present for a given patient note. The attribute extractions of these notes were then verified for by authors of this paper. Extraction 1 Prompt: For more details on step 1, we provided a set of 32 parameters which cover at least one attribute needed for each of the 55 calculators. For each note in Open-Patients, we applied the prompt shown below to determine which of the 32 parameters could be extracted from each note.