Interpretable Neural Networks for Predicting Mortality Risk using Multi-modal Electronic Health Records

Abstract--We present an interpretable neural network for predicting an important clinical outcome (1-year mortality) from multi-modal Electronic Health Record (EHR) data. Our approach buildson prior multi-modal machine learning models by now enabling visualization of how individual factors contribute to the overall outcome risk, assuming other factors remain constant, which was previously impossible. We demonstrate the value of this approach using a large multi-modal clinical dataset including both EHR data and 31,278 echocardiographic videos of the heart from 26,793 patients. We generated separate models for (i) clinical data only (CD) (e.g. The interpretable multi-modal model maintained performance compared to non-interpretable models (Random Forest, XG-Boost), and also performed significantly better than a model using a single modality (average AUC 0.82). Clinically relevant insights and multi-modal variable importance rankings were also facilitated by the new model, which have previously been impossible. I. INTRODUCTION The adoption of Electronic Health Records (EHR) in medicine has facilitated the collection of massive amounts of clinical data which can be used to develop highly accurate risk models that physicians can use to guide medical decision making. To take full advantage of the available EHR data, these models, similar to a physician, need to be able to handle multiple modalities as inputs. For example, both tabular data such as laboratory measurements and pixel data from clinical images should be readily incorporated. This basic framework is shown in Figure 1. As documented in [1], [2], [3], [4], precision medicine can benefit greatly from development of these risk models. The proliferation of these models has prompted scrutiny from the medical community, which demands clinical validity and interpretability to improve usefulness [5] and inclusion of all relevant predictors (or, conversely, explanation when a relevant data input is excluded) [6]. Thus, any medical risk model should be interpretable and facilitate understanding of the various contributions of different inputs towards the overall risk assessment.