HyMNet: a Multimodal Deep Learning System for Hypertension Classification using Fundus Photographs and Cardiometabolic Risk Factors

In recent years, deep learning has shown promise in predicting hypertension (HTN) from fundus images. However, most prior research has primarily focused on analyzing a single type of data, which may not capture the full complexity of HTN risk. To address this limitation, this study introduces a multimodal deep learning (MMDL) system, dubbed HyMNet, which combines fundus images and cardiometabolic risk factors, specifically age and gender, to improve hypertension detection capabilities. Our MMDL system uses the DenseNet-201 architecture, pre-trained on ImageNet, for the fundus imaging path and a fully connected neural network for the age and gender path. The two paths are jointly trained by concatenating 64 features output from each path that are then fed into a fusion network. The system was trained on 1,143 retinal images from 626 individuals collected from the Saudi Ministry of National Guard Health Affairs. The results show that the multimodal model that integrates fundus images along with age and gender achieved an AUC of 0.791 [CI: 0.735, 0.848], which outperforms the unimodal model trained solely on fundus photographs that yielded an AUC of 0.766 [CI: 0.705, 0.828] for hypertension detection. Abbreviations BP, blood pressure; CVD, cardiovascular disease; EHR, electronic health record; EMR, electronic medical records; AI, artificial intelligence; DL, deep learning; MMDL, multimodal deep learning; SVM, support vector machine; FCNN, fully connected neural network; CNN convolutional neural network; ReLU; rectified linear unit; AUC, area under the operating characteristic curve, PR, area under the precision-recall curve; CI, confidence interval; MAE, mean absolute error; KAIMRC, King Abdullah International Medical Research Center. Keywords Artificial Intelligence; Machine Learning; Computer Vision. 1. Introduction Cardiovascular diseases persist as one of the primary causes of mortality worldwide, with hypertension, or high blood pressure (BP), serving as a significant contributing risk factor (1,2).