Explainable AI for Clinical Outcome Prediction: A Survey of Clinician Perceptions and Preferences

Explainable AI for Clinical Outcome Prediction: A Survey of Clinician Perceptions and Preferences Jun Hou, MS 1, Lucy Lu Wang, PhD 2 1 Virginia T ech, Blacksburg, V A; 2 University of Washington, Seattle, W A Abstract Explainable AI (XAI) techniques are necessary to help clinicians make sense of AI predictions and integrate predictions into their decision-making workflow. In this work, we conduct a survey study to understand clinician preference among different XAI techniques when they are used to interpret model predictions over text-based EHR data. We implement four XAI techniques (LIME, Attention-based span highlights, exemplar patient retrieval, and free-text rationales generated by LLMs) on an outcome prediction model that uses ICU admission notes to predict a patient's likelihood of experiencing in-hospital mortality. Using these XAI implementations, we design and conduct a survey study of 32 practicing clinicians, collecting their feedback and preferences on the four techniques. We synthesize our findings into a set of recommendations describing when each of the XAI techniques may be more appropriate, their potential limitations, as well as recommendations for improvement. I NTRODUCTION Clinical decision support systems (CDSS) powered by machine learning and AI have the potential to assist in medical decisions and improve patient outcomes. However, to meaningfully support clinicians, AI-powered CDSS must be trustworthy and interpretable, allowing clinicians to assess the utility and applicability of model predictions. Explainable AI (XAI) techniques have been proposed to improve model interpretability, especially for neural network and other blackbox models. 1 While XAI techniques have been applied to CDSS, 2 a comprehensive understanding of clinician preferences and perceptions regarding XAI applications in these systems remains largely unexplored. Prior work on clinical XAI tends to focus on explanatory accuracy, in terms of which models are applicable, 3 how to integrate XAI methods for different healthcare tasks, 4 or which datasets are available to train on.