MLCA-AVSR: Multi-Layer Cross Attention Fusion based Audio-Visual Speech Recognition

While automatic speech recognition (ASR) systems degrade significantly Following this, plenty of studies have adopted a cross-attention module in noisy environments, audio-visual speech recognition to capture inherent alignments and complementary information (AVSR) systems aim to complement the audio stream with noiseinvariant between fully encoded audio-visual representations [9, 10, 11]. Additionally, visual cues and improve the system's robustness. However, some works directly concatenate the raw speech and video current studies mainly focus on fusing the well-learned modality sequences together and employ a shared encoder with self-attention features, like the output of modality-specific encoders, without mechanisms to learn modality alignments [2, 12]. In [13, 14], considering the contextual relationship during the modality feature hidden features from different layers of audio and visual encoders learning. In this study, we propose a multi-layer cross-attention were leveraged to achieve more effective fusion, indicating that conducting fusion based AVSR (MLCA-AVSR) approach that promotes representation multi-layer fusion can promote the performance of AVSR learning of each modality by fusing them at different levels systems. of audio/visual encoders. Experimental results on the MISP2022-Recently, the Multi-modal Information based Speech Processing AVSR Challenge dataset show the efficacy of our proposed system, (MISP) Challenge series [15, 16, 17] has been introduced to achieving a concatenated minimum permutation character error rate explore the utilization of both audio and visual data in distant multimicrophone (cpCER) of 30.57% on the Eval set and yielding up to 3.17% relative signal processing tasks, like keyword spotting and improvement compared with our previous system which ranked speech recognition.