Goto

Collaborating Authors

 Information Fusion


TMI-CLNet: Triple-Modal Interaction Network for Chronic Liver Disease Prognosis From Imaging, Clinical, and Radiomic Data Fusion

arXiv.org Artificial Intelligence

Chronic liver disease represents a significant health challenge worldwide and accurate prognostic evaluations are essential for personalized treatment plans. Recent evidence suggests that integrating multimodal data, such as computed tomography imaging, radiomic features, and clinical information, can provide more comprehensive prognostic information. However, modalities have an inherent heterogeneity, and incorporating additional modalities may exacerbate the challenges of heterogeneous data fusion. Moreover, existing multimodal fusion methods often struggle to adapt to richer medical modalities, making it difficult to capture inter-modal relationships. To overcome these limitations, We present the Triple-Modal Interaction Chronic Liver Network (TMI-CLNet). Specifically, we develop an Intra-Modality Aggregation module and a Triple-Modal Cross-Attention Fusion module, which are designed to eliminate intra-modality redundancy and extract cross-modal information, respectively. Furthermore, we design a Triple-Modal Feature Fusion loss function to align feature representations across modalities. Extensive experiments on the liver prognosis dataset demonstrate that our approach significantly outperforms existing state-of-the-art unimodal models and other multi-modal techniques. Our code is available at https://github.com/Mysterwll/liver.git.


EKF-Based Radar-Inertial Odometry with Online Temporal Calibration

arXiv.org Artificial Intelligence

Accurate time synchronization between heterogeneous sensors is crucial for ensuring robust state estimation in multi-sensor fusion systems. Sensor delays often cause discrepancies between the actual time when the event was captured and the time of sensor measurement, leading to temporal misalignment (time offset) between sensor measurement streams. In this paper, we propose an extended Kalman filter (EKF)-based radar-inertial odometry (RIO) framework that estimates the time offset online. The radar ego-velocity measurement model, estimated from a single radar scan, is formulated to include the time offset for the update. By leveraging temporal calibration, the proposed RIO enables accurate propagation and measurement updates based on a common time stream. Experiments on multiple datasets demonstrated the accurate time offset estimation of the proposed method and its impact on RIO performance, validating the importance of sensor time synchronization. Our implementation of the EKF-RIO with online temporal calibration is available at https://github.com/spearwin/EKF-RIO-TC.


Multimodal Magic Elevating Depression Detection with a Fusion of Text and Audio Intelligence

arXiv.org Artificial Intelligence

ABSTRACT This study proposes an innovative multimodal fusion model based on a teacherstudent architecture to enhance the accuracy of depression classification. Our designed model addresses the limitations of traditional methods in feature fusion and modality weight allocation by introducing multi-head attention mechanisms and weighted multimodal transfer learning. Leveraging the DAIC-WOZ dataset, the student fusion model, guided by textual and auditory teacher models, achieves significant improvements in classification accuracy. Ablation experiments demonstrate that the proposed model attains an F1 score of 99. 1% on the test set, significantly outperforming unimodal and conventional approaches. Our method effectively captures the complementarity between textual and audio features while dynamically adjusting the contributions of the teacher models to enhance generalization capabilities. The experimental results highlight the robustness and adaptability of the proposed framework in handling complex multimodal data. This research provides a novel technical framework for multimodal large model learning in depression analysis, offering new insights into addressing the limitations of existing methods in modality fusion and feature extraction. INTRODUCTION Depression is a significant global health concern that affects millions of individuals across various demographics, leading to considerable social, economic, and health-related impacts. According to the World Health Organization (WHO), depression is one of the leading causes of disability worldwide, with over 264 million people affected.


An Optimal Cascade Feature-Level Spatiotemporal Fusion Strategy for Anomaly Detection in CAN Bus

arXiv.org Artificial Intelligence

Autonomous vehicles represent a revolutionary advancement driven by the integration of artificial intelligence within intelligent transportation systems. However, they remain vulnerable due to the absence of robust security mechanisms in the Controller Area Network (CAN) bus. In order to mitigate the security issue, many machine learning models and strategies have been proposed, which primarily focus on a subset of dominant patterns of anomalies and lack rigorous evaluation in terms of reliability and robustness. Therefore, to address the limitations of previous works and mitigate the security vulnerability in CAN bus, the current study develops a model based on the intrinsic nature of the problem to cover all dominant patterns of anomalies. To achieve this, a cascade feature-level fusion strategy optimized by a two-parameter genetic algorithm is proposed to combine temporal and spatial information. Subsequently, the model is evaluated using a paired t-test to ensure reliability and robustness. Finally, a comprehensive comparative analysis conducted on two widely used datasets advocates that the proposed model outperforms other models and achieves superior accuracy and F1-score, demonstrating the best performance among all models presented to date.


Targeted Data Fusion for Causal Survival Analysis Under Distribution Shift

arXiv.org Machine Learning

Causal inference across multiple data sources has the potential to improve the generalizability, transportability, and replicability of scientific findings. However, data integration methods for time-to-event outcomes -- common in medical contexts such as clinical trials -- remain underdeveloped. Existing data fusion methods focus on binary or continuous outcomes, neglecting the distinct challenges of survival analysis, including right-censoring and the unification of discrete and continuous time frameworks. To address these gaps, we propose two novel approaches for multi-source causal survival analysis. First, considering a target site-specific causal effect, we introduce a semiparametric efficient estimator for scenarios where data-sharing is feasible. Second, we develop a federated learning framework tailored to privacy-constrained environments. This framework dynamically adjusts source site-specific contributions, downweighting biased sources and upweighting less biased ones relative to the target population. Both approaches incorporate nonparametric machine learning models to enhance robustness and efficiency, with theoretical guarantees applicable to both continuous and discrete time-to-event outcomes. We demonstrate the practical utility of our methods through extensive simulations and an application to two randomized trials of a monoclonal neutralizing antibody for HIV-1 prevention: HVTN 704/HPTN 085 (cisgender men and transgender persons in the Americas and Switzerland) and HVTN 703/HPTN 081 (women in sub-Saharan Africa). The results highlight the potential of our approaches to efficiently estimate causal effects while addressing heterogeneity across data sources and adhering to privacy and robustness constraints.


Chinese Stock Prediction Based on a Multi-Modal Transformer Framework: Macro-Micro Information Fusion

arXiv.org Artificial Intelligence

This paper proposes an innovative Multi-Modal Transformer framework (MMF-Trans) designed to significantly improve the prediction accuracy of the Chinese stock market by integrating multi-source heterogeneous information including macroeconomy, micro-market, financial text, and event knowledge. The framework consists of four core modules: (1) A four-channel parallel encoder that processes technical indicators, financial text, macro data, and event knowledge graph respectively for independent feature extraction of multi-modal data; (2) A dynamic gated cross-modal fusion mechanism that adaptively learns the importance of different modalities through differentiable weight allocation for effective information integration; (3) A time-aligned mixed-frequency processing layer that uses an innovative position encoding method to effectively fuse data of different time frequencies and solves the time alignment problem of heterogeneous data; (4) A graph attention-based event impact quantification module that captures the dynamic impact of events on the market through event knowledge graph and quantifies the event impact coefficient. We introduce a hybrid-frequency Transformer and Event2Vec algorithm to effectively fuse data of different frequencies and quantify the event impact. Experimental results show that in the prediction task of CSI 300 constituent stocks, the root mean square error (RMSE) of the MMF-Trans framework is reduced by 23.7% compared to the baseline model, the event response prediction accuracy is improved by 41.2%, and the Sharpe ratio is improved by 32.6%.


Reviews: Kalman Filter, Sensor Fusion, and Constrained Regression: Equivalences and Insights

Neural Information Processing Systems

Rebuttal acknowledged, thank you for the additional clarifications. Indeed, given a flat prior for x_{t 1} (i.e., Gaussian with "infinite" variance), we have two independent observations: - the influence of the past (prediction term) - the influence of the current measurement (filtering term) both have Gaussian likelihood. So the posterior density of x_{t 1} is proportional to a product of three Gaussian-shaped terms. The two different ways in which these terms can be folded into each other (using standard Gaussian conjugacy rules) lead to Thm 1. I believe that the linear-algebraic formulation the authors use just hides the fact that we are multiplying Gaussian PDFs in different ways.


Reviews: Kalman Filter, Sensor Fusion, and Constrained Regression: Equivalences and Insights

Neural Information Processing Systems

The results of this paper could be of interest to NeurIPS. However, the author(s) should try and address most of the concerns raised by the reviewers, specially a review of existing work on Kalman filtering with infinite variance.


Reviews: On Single Source Robustness in Deep Fusion Models

Neural Information Processing Systems

Summary This paper discusses the importance and the method for deep fusion model with single-source noise with experiments on 3D/BEV object detection. It first proposes a novel loss called MAXSSN, as a loss used in the whole paper for single-source robustness. It then shows the limitation of standard robust fusion model -- if we do not consider every single loss separately -- adding all of them to the input at once, we would get a worse model. Two algorithms are proposed for minimizing the MAXSSN loss. The basic idea is to alternatively train on clean data and data with noise.


Reviews: On Single Source Robustness in Deep Fusion Models

Neural Information Processing Systems

This is a borderline paper that discusses the problem of robustness with two sources. The reviewers thought the approach was novel; however, there is a lack of theoretical analysis and question about the method's generalization to more complex scenarios.