A Network-based Multimodal Data Fusion Approach for Characterizing Dynamic Multimodal Physiological Patterns

Abstract-- Characterizing the dynamic interactive patterns of complex systems helps gain in-depth understanding of how components interrelate with each other while performing certain functions as a whole. In this study, we present a novel multimodal data fusion approach to construct a complex network, which models the interactions of biological subsystems in the human body under emotional states through physiological responses. Joint recurrence plot and temporal network metrics are employed to integrate the multimodal information at the signal level. A benchmark public dataset of is used for evaluating our model. I. INTRODUCTION Daily activities of human body are performed through the joint functioning of biological subsystems, including nervous, muscular, respiratory, etc. Extensive attention has been devoted into developing methods for utilizing the rich information collected from human body via multiple sources, while each source of information is referred to as a modality.