Exploring a Multimodal Fusion-based Deep Learning Network for Detecting Facial Palsy

Facial palsy has serious consequences on patients, such as diminished feeding function, psychological distress, and social withdrawal [10]. For diagnosis of facial palsy, clinicians usually conduct observation-based physical examinations [7]. However, it is challenging to quantify symptom intensity and variation, measure changes in these symptoms between visits for a single patient, and measure differences in symptoms across different patients at the same time [8]. To address this challenge, researchers have explored various algorithmic approaches to detect facial palsy [8, 9, 15, 17]. These approaches broadly fall into two categories: 1) those employing machine learning models with manual feature extraction and 2) those that leverage deep learning-based models. For approaches with manual features, Ngo et al. [15] proposed a frequency-based technique using limited-orientation modified circular Gabor filters (LO-MCGFS) to magnify desired frequencies in dataset images and extract features from rotation invariant texture regions for classifying facial palsy. In addition, researchers explored to train a data-driven model [6, 9, 17] to detect facial key points and computed features, such as the displacement ratio between left and right halves of the face or the motion information of facial regions. Alternatively, researchers discussed the limitations of using manual features and leveraged RGB images or images with facial line segments to train deep learning-based models (e.g.