Multi-Modal Contrastive Learning for Online Clinical Time-Series Applications

Electronic Health Record (EHR) datasets from Intensive Care Units (ICU) contain a diverse set of data modalities. While prior works have successfully leveraged multiple modalities in supervised settings, we apply advanced self-supervised multi-modal contrastive learning techniques to ICU data, specifically focusing on clinical notes and time-series for clinically relevant online prediction tasks. We introduce a loss function Multi-Modal Neighborhood Contrastive Loss (MM-NCL), a soft neighborhood function, and showcase the excellent linear probe and zeroshot performance of our approach. Electronic Health Record (EHR) data from Intensive Care Units (ICUs) has emerged as a valuable resource for predicting clinically relevant quantities in recent years (Hyland et al., 2020; Hüser et al., 2024; Yèche et al., 2022; Pace et al., 2022). However, the diverse nature of EHR data, encompassing different modalities such as clinical notes and time series, presents a challenge for effective utilization.