Progressive Multi-Source Domain Adaptation for Personalized Facial Expression Recognition

Abstract--Personalized facial expression recognition (FER) involves adapting a machine learning model using samples from labeled sources and unlabeled target domains. Given the challenges of recognizing subtle expressions with considerable interpersonal variability, state-of-the-art unsupervised domain adaptation (UDA) methods focus on the multi-source UDA (MSDA) setting, where each domain corresponds to a specific subject, and improve model accuracy and robustness. State-of-the-art MSDA methods for FER address this domain shift by considering all the sources to adapt to the target representations. Nevertheless, adapting to a target subject presents significant challenges due to large distributional differences between source and target domains, often resulting in negative transfer . In addition, integrating all sources simultaneously increases computational costs and causes misalignment with the target. T o address these issues, we propose a progressive MSDA approach that gradually introduces information from subjects (source domains) based on their similarity to the target subject. This will ensure that only the most relevant sources from the target are selected, which helps avoid the negative transfer caused by dissimilar sources. During adaptation, the source domains are introduced in a curriculum manner . We first exploit the closest sources to reduce the distribution shift with the target and then move towards the furthest while only considering the most relevant sources based on the predetermined threshold. Furthermore, to mitigate catastrophic forgetting caused by the incremental introduction of source subjects, we implemented a density-based memory mechanism that preserves the most relevant historical source samples for adaptation. Further, performance is evaluated on a cross-dataset setting (UNBC-McMaster BioVid), showing the importance of gradually adapting to source subjects. N recent years, there has been a growing demand for deep learning (DL) models that can perform well on FER across various industrial sectors such as in detecting suspicious or criminal behavior, automated emotion recognition, or the estimation of pain in health care [1]-[4]. The authors are affiliated with the LIVIA and ILLS, the Department of Systems Engineering, and the Department of Software Engineering at ETS Montreal, Canada. Therefore, adapting a deep FER model to a specific individual (i.e., personalization) is important to maintain a high level of performance. Personalized FER has been extensively studied in the literature, primarily through supervised learning approaches and fine-tuning techniques [6]-[8] to capture individual-specific nuances. These approaches mostly rely on fully or weakly labeled data to adapt and create a personalized model for each subject.