MLLM-based Speech Recognition: When and How is Multimodality Beneficial?

MLLM-based Speech Recognition: When and How is Multimodality Beneficial? Abstract--Recent advances in multi-modal large language models (MLLMs) have opened new possibilities for unified modeling of speech, text, images, and other modalities. Bui lding on our prior work [1], this paper examines the conditions and model architectures under which multiple input modalities can improve automatic speech recognition (ASR) accuracy in noi sy environments. Through experiments on synthetic and real-w orld data, we find that (1) harnessing more modalities usually improves ASR accuracy, as each modality provides complementa ry information, but the improvement depends on the amount of auditory noise. These findings b oth offer practical insights and help to deepen our understandi ng of multi-modal speech recognition under challenging conditi ons. IVEN the success of large language models (LLMs) for natural language processing as enabled by their reasoning and contextual understanding abilities, resear chers are increasingly exploring how to develop multi-modal LLMs (MLLMs) to harness multiple input modalities, particularl y in areas involving speech and vision [2-4]. The evolution of LLMs - specifically defined as large-scale autoregressive decoder-only models - has led to the emergence of LLMbased automatic speech recognition (ASR) systems in the pas t few years [5-7]. LLM-based ASR models typically adopt decoder-only architectures, taking either discrete units (e.g., extracted from HuBERT [8]) or continuous features (e.g., Lo g-Mel filterbank) as input, and producing text transcriptions as output. While conventional speech processing methods consider only the audio itself, MLLM-based speech recognizers jointly model and process multiple input modalities, such a s subject-matter context or visual cues like images and the speaker's lip movements [9-11]. "door" and "dough", while lip movements help differentiate between "night" and "right"; or like the example in Figure 1 that an image of a man in a fedora can help clarify the descriptive speech. They also benefit from LLM pre-training which endows them with linguistic information about which transcripts are more or less likely in a given context. MLLM-based approaches hold significant promise for next-generation ASR systems but they also bring new challenges: While multiple input modalities may contain complementary information, they also increase the sequence length and com - plexity, making modeling more difficult.