Multimodal LLM Integrated Semantic Communications for 6G Immersive Experiences

In connected autonomous driving, the edge infrastructure delivers global environmental awareness to compensate for vehicle occlusions. Through the proposed MLLM-SC framework, the MLLM processes multimodal inputs from roadside cameras, LiDARs, and queries of vehicles to understand the complex traffic scenes and reason about vehicle data requirements. Then, the MLLM provides guidance to the semantic communication module to prioritize encoding and transmission of multimodal sensor data in critical occluded areas. Highly important safety information receives dedicated high-reliability channels, while non-occluded data undergoes bandwidth-efficient compression, ensuring life-critical information reaches autonomous vehicles with minimal latency through intelligent scene understanding and visual reasoning capabilities. B. Key Challenges (1) Efficient Semantic Alignment Across Heterogeneous Modalities. In multimodal communication scenarios such as human-machine interaction and autonomous systems, diverse task requests, including eye-tracking signals, voice and textual commands, can be jointly interpreted to support task-oriented transmission. We adopt semantic alignment techniques that project heterogeneous modalities into a shared representation space like Contrastive Language-Image Pre-training (CLIP) [9], which align visual and textual data through contrastive learning. Effective mapping and alignment within a shared semantic space is fundamental to supporting downstream inference and task-oriented transmission.