FAF: A novel multimodal emotion recognition approach integrating face, body and text

How to improve the accuracy of emotion recognition has become a primary issue. In recent years, with the continuous development of artificial intelligence technology, human-computer interaction has become the focus of research in the field of information science. As one of the critical technologies to realize human-computer interaction, emotion recognition has gradually received a lot of attention from researchers. At present, most of the research works on emotion recognition are based on single-modal, such as facial expressions [1-3], body movements [4-5] and speech text [6-7]. However, emotion recognition based on unimodal often has limitations and, in most cases, could only reflect a portion of human emotional expression. Multimodal emotion recognition can link individual unimodal channels and use the feature complementarity between channels to combine multiple information to determine the emotional state. Studies have shown that the multimodal emotion recognition approach has better performance than unimodal emotion judgment in most cases [8]. The difficulty of multimodal recognition is not only to control the internal information of individual modality (Intra-modality), but also to complement the interactive features between individual modalities (Inter-modality). It has been extensively studied by scholars, such as Tensor Fusion Network (TFN) proposed by Zadeh et al [9], Polynomial Tensor Pooling (PTP) proposed by Hou et al [10], and Memory Fusion Network (MFN) presented by Zadeh et al [11].