Performance of Large Language Models in Answering Critical Care Medicine Questions

Abstract: Large Language Models have been tested on medical student-level questions, but their performance in specialized fields like Critical Care Medicine (CCM) is less explored. This study evaluated Meta-Llama 3.1 models (8B and 70B parameters) on 871 CCM questions. Performance varied across domains, highest in Research (68.4%) and lowest in Renal (47.9%), highlighting the need for broader future work to improve models across various subspecialty domains. Introduction: The use of Large Language Models (LLMs) to answer medical exam - style questions has gained popularity in recent years. This study aims to evaluate the performance of LLMs in answering subspecialty CCM board exam - style questions.