Multimodal Fusion at Three Tiers: Physics-Driven Data Generation and Vision-Language Guidance for Brain Tumor Segmentation

Accurate brain tumor segmentation is crucial for neuro-oncology diagnosis and treatment planning. Deep learning methods have made significant progress, but automatic segmentation still faces challenges. These include tumor morphological heterogeneity and complex three-dimensional spatial relationships. This paper proposes a three-tier fusion architecture that achieves precise brain tumor segmentation. The method processes information progressively at the pixel, feature, and semantic levels. At the pixel level, physical modeling extends magnetic resonance imaging (MRI) to multimodal data. This includes simulated ultrasound and synthetic computed tomography (CT). At the feature level, the method achieves Transformer-based cross-modal feature fusion through multi-teacher collaborative distillation. At the semantic level, clinical textual knowledge generated by GPT-4V transforms into spatial guidance signals. This transformation uses Contrastive Language-Image Pre-training (CLIP) contrastive learning and Feature-wise Linear Modulation (FiLM). These three tiers work together to form a complete processing chain. The chain spans from data augmentation to feature extraction to semantic guidance. The model achieves average Dice coefficients of 0.8665, 0.9014, and 0.8912 on the three datasets, respectively. The 95% Hausdorff Distance (HD95) reduces by an average of 6.57 millimeters compared to the baseline. Introduction Magnetic resonance imaging (MRI) segmentation of brain tumors is fundamental for neuro-oncology diagnosis and treatment planning. Different sub-regions have different prognostic significance.