Goto

Collaborating Authors

 Deep Learning


Do Video Language Models Really Know Where to Look? Diagnosing Attention Failures in Video Language Models

arXiv.org Artificial Intelligence

Recent advances in multimodal large language models (MLLMs) have led to much progress in video understanding tasks. To avoid the heavy computational cost of processing all frames, these models typically rely on keyframe sampling methods guided by vision-language encoders (\textit{e.g.,} SigLIP). However, it remains unclear whether such encoders can truly identify the most informative frames. In this work, we provide several empirical pieces of evidence revealing that popular vision encoders critically suffer from their limited capability to identify where the MLLM should look inside the video to handle the given textual query appropriately. Our findings suggest that the development of better keyframe identification techniques may be necessary for efficient video MLLMs.


A Multimodal Deep Learning Framework for Early Diagnosis of Liver Cancer via Optimized BiLSTM-AM-VMD Architecture

arXiv.org Artificial Intelligence

This paper proposes a novel multimodal deep learning framework integrating bidirectional LSTM, multi-head attention mechanism, and variational mode decomposition (BiLSTM-AM-VMD) for early liver cancer diagnosis. Using heterogeneous data that include clinical characteristics, biochemical markers, and imaging-derived variables, our approach improves both prediction accuracy and interpretability. Experimental results on real-world datasets demonstrate superior performance over traditional machine learning and baseline deep learning models.


Multi-Modal Machine Learning Framework for Predicting Early Recurrence of Brain Tumors Using MRI and Clinical Biomarkers

arXiv.org Artificial Intelligence

Despite advances in neurosurgical techniques, radiation therapy, and chemotherapeutic regimens, the prognosis for patients remains poor, with recurrence rates exceeding 70% within two years after surgical resection [70]. The high probability of early recurrence underscores the critical need for robust, patient-specific prognostic tools that can support personalized clinical decision-making. Accurate prediction of tumor recurrence is crucial for stratifying patients into appropriate risk categories, guiding adjuvant therapy choices, and optimizing postoperative surveillance [71]. Traditionally, recurrence risk assessments are based on clinical judgment informed by factors such as tumor size, histological grade, and extent of resection. However, these assessments often lack the granularity and precision needed for an individualized prognosis, particularly given the heterogeneity of tumor biology and interpatient variability [72]. Recent advances in multimodal machine learning (ML) have demonstrated the potential to address these challenges by integrating diverse data sources, as evidenced by emerging frameworks that leverage collaborative model optimization [73-75] and reward-driven paradigms for multimodal fusion [76,77]. Structural magnetic resonance imaging (MRI), especially contrast-enhanced T1-weighted imaging, is routinely employed in brain tumor diagnosis and follow-up, providing insights into tumor morphology and contrast uptake behavior [78].


Zero-shot Cross-lingual NER via Mitigating Language Difference: An Entity-aligned Translation Perspective

arXiv.org Artificial Intelligence

Cross-lingual Named Entity Recognition (CL-NER) aims to transfer knowledge from high-resource languages to low-resource languages. However, existing zero-shot CL-NER (ZCL-NER) approaches primarily focus on Latin script language (LSL), where shared linguistic features facilitate effective knowledge transfer. In contrast, for non-Latin script language (NSL), such as Chinese and Japanese, performance often degrades due to deep structural differences. To address these challenges, we propose an entity-aligned translation (EAT) approach. Leveraging large language models (LLMs), EAT employs a dual-translation strategy to align entities between NSL and English. In addition, we fine-tune LLMs using multilingual Wikipedia data to enhance the entity alignment from source to target languages.


Dream-Coder 7B: An Open Diffusion Language Model for Code

arXiv.org Artificial Intelligence

We present Dream-Coder 7B, an open-source discrete diffusion language model for code generation that exhibits emergent any-order generation capabilities. Unlike traditional autoregressive (AR) models that decode strictly left-to-right, Dream-Coder 7B adaptively determines its decoding strategy based on the coding task: sketch-first generation for complex algorithms, left-to-right generation for straightforward completions, and interleaved reasoning generation for code understanding tasks. We adapt a pretrained AR checkpoint to a discrete diffusion frameworks with a continuous-time weighted cross-entropy objective. Our post-training recipe comprises (i) supervised fine-tuning, where we mitigate padding pathologies via random truncation and a padding penalty to improve sample efficiency and stabilize generation; and (ii) reinforcement learning with verifiable rewards over a curated high-quality prompt set drawn from open-source datasets, using a tailored reinforcement learning recipe for diffusion language models. The resulting Dream-Coder 7B Instruct attains 21.4\% pass@1 on LiveCodeBench (2410--2505) and demonstrates competitive performance on HumanEval, MBPP, BigCodeBench, and CRUXEval. We release Dream-Coder-7B and Dream-Coder-7B-Instruct checkpoints, training recipes, preprocessing pipelines, and inference code to facilitate reproducibility and further research.


MATL-DC: A Multi-domain Aggregation Transfer Learning Framework for EEG Emotion Recognition with Domain-Class Prototype under Unseen Targets

arXiv.org Artificial Intelligence

Emotion recognition based on electroencephalography (EEG) signals is increasingly becoming a key research hotspot in affective Brain-Computer Interfaces (aBCIs). However, the current transfer learning model greatly depends on the source domain and target domain data, which hinder the practical application of emotion recognition. Therefore, we propose a Multi-domain Aggregation Transfer Learning framework for EEG emotion recognition with Domain-Class prototype under unseen targets (MATL-DC). We design the feature decoupling module to decouple class-invariant domain features from domain-invariant class features from shallow features. In the model training stage, the multi-domain aggregation mechanism aggregates the domain feature space to form a superdomain, which enhances the characteristics of emotional EEG signals. In each superdomain, we further extract the class prototype representation by class features. In addition, we adopt the pairwise learning strategy to transform the sample classification problem into the similarity problem between sample pairs, which effectively alleviates the influence of label noise. It is worth noting that the target domain is completely unseen during the training process. In the inference stage, we use the trained domain-class prototypes for inference, and then realize emotion recognition. We rigorously validate it on the publicly available databases (SEED, SEED-IV and SEED-V). The results show that the accuracy of MATL-DC model is 84.70\%, 68.11\% and 61.08\%, respectively. MATL-DC achieves comparable or even better performance than methods that rely on both source and target domains. The source code is available at https://github.com/WuCB-BCI/MATL-DC.


SC-GIR: Goal-oriented Semantic Communication via Invariant Representation Learning

arXiv.org Artificial Intelligence

Goal-oriented semantic communication (SC) aims to revolutionize communication systems by transmitting only task-essential information. However, current approaches face challenges such as joint training at transceivers, leading to redundant data exchange and reliance on labeled datasets, which limits their task-agnostic utility. To address these challenges, we propose a novel framework called Goal-oriented Invariant Representation-based SC (SC-GIR) for image transmission. Our framework leverages self-supervised learning to extract an invariant representation that encapsulates crucial information from the source data, independent of the specific downstream task. This compressed representation facilitates efficient communication while retaining key features for successful downstream task execution. Focusing on machine-to-machine tasks, we utilize covariance-based contrastive learning techniques to obtain a latent representation that is both meaningful and semantically dense. To evaluate the effectiveness of the proposed scheme on downstream tasks, we apply it to various image datasets for lossy compression. The compressed representations are then used in a goal-oriented AI task. Extensive experiments on several datasets demonstrate that SC-GIR outperforms baseline schemes by nearly 10%,, and achieves over 85% classification accuracy for compressed data under different SNR conditions. These results underscore the effectiveness of the proposed framework in learning compact and informative latent representations.


Natural Context Drift Undermines the Natural Language Understanding of Large Language Models

arXiv.org Artificial Intelligence

How does the natural evolution of context paragraphs affect question answering in generative Large Language Models (LLMs)? To investigate this, we propose a framework for curating naturally evolved, human-edited variants of reading passages from contemporary QA benchmarks and for analyzing LLM performance across a range of semantic similarity scores, which quantify how closely each variant aligns with content seen during pretraining. Using this framework, we evaluate six QA datasets and eight LLMs with publicly available training data. Our experiments reveal that LLM performance declines as reading passages naturally diverge from the versions encountered during pretraining-even when the question and all necessary information remains present at inference time. For instance, average model accuracy on BoolQ drops by over 30% from the highest to lowest similarity bins, with slopes exceeding 70 across several LLMs. These findings suggest that natural text evolution poses a significant challenge to the language understanding capabilities of LLMs.


IMU-Enhanced EEG Motion Artifact Removal with Fine-Tuned Large Brain Models

arXiv.org Artificial Intelligence

-- Electroencephalography (EEG) is a non-invasive method for measuring brain activity with high temporal resolution; however, EEG signals often exhibit low signal-to-noise ratios because of contamination from physiological and environmental artifacts. One of the major challenges hindering the real-world deployment of brain-computer interfaces (BCIs) involves the frequent occurrence of motion-related EEG artifacts. Most prior studies on EEG motion artifact removal rely on single-modality approaches, such as Artifact Subspace Reconstruction (ASR) and Independent Component Analysis (ICA), without incorporating simultaneously recorded modalities like inertial measurement units (IMUs), which directly capture the extent and dynamics of motion. This work proposes a fine-tuned large brain model (LaBraM)-based correlation attention mapping method that leverages spatial channel relationships in IMU data to identify motion-related artifacts in EEG signals. The fine-tuned model contains approximately 9.2 million parameters and uses 5.9 hours of EEG and IMU recordings for training, just 0.2346% of the 2500 hours used to train the base model. We compare our results against the established ASR-ICA benchmark across varying time scales and motion activities, showing that incorporating IMU reference signals significantly improves robustness under diverse motion scenarios.


DRetNet: A Novel Deep Learning Framework for Diabetic Retinopathy Diagnosis

arXiv.org Artificial Intelligence

Diabetic retinopathy (DR) is a leading cause of blindness worldwide, necessitating early detection to prevent vision loss. Current automated DR detection systems often struggle with poor-quality images, lack interpretability, and insufficient integration of domain-specific knowledge. To address these challenges, we introduce a novel framework that integrates three innovative contributions: (1) Adaptive Retinal Image Enhancement Using Physics-Informed Neural Networks (PINNs): this technique dynamically enhances retinal images by incorporating physical constraints, improving the visibility of critical features such as microaneurysms, hemorrhages, and exudates; (2) Hybrid Feature Fusion Network (HFFN): by combining deep learning embeddings with handcrafted features, HFFN leverages both learned representations and domain-specific knowledge to enhance generalization and accuracy; (3) Multi-Stage Classifier with Uncertainty Quantification: this method breaks down the classification process into logical stages, providing interpretable predictions and confidence scores, thereby improving clinical trust. The proposed framework achieves an accuracy of 92.7%, a precision of 92.5%, a recall of 92.6%, an F1-score of 92.5%, an AUC of 97.8%, a mAP of 0.96, and an MCC of 0.85. Ophthalmologists rated the framework's predictions as highly clinically relevant (4.8/5), highlighting its alignment with real-world diagnostic needs. Qualitative analyses, including Grad-CAM visualizations and uncertainty heatmaps, further enhance the interpretability and trustworthiness of the system. The framework demonstrates robust performance across diverse conditions, including low-quality images, noisy data, and unseen datasets. These features make the proposed framework a promising tool for clinical adoption, enabling more accurate and reliable DR detection in resource-limited settings.