Multimodal Federated Learning in Healthcare: a review

Bigger picture Building efficient multimodal AI algorithms from biomedical data, including medical imaging, electronic health records, and personalized sensors would have immense utility for a broad range of health applications. An increasingly common approach towards building such algorithms is a centralized framework, where all the training databases are held together for training models. However, especially in healthcare, due to variety of reasons, including privacy, security, and ethical reasons, gathering datasets from different hospitals, and health centers is not feasible. Federated learning has emerged as an elegant alternative to centralized machine learning where models are trained separately and collaboratively without a centralized database. However, must of the federated learning models have shown their efficacy in the unimodal system, and there is an increasing need for advanced FL algorithms to allow the training of larger and capable model that can absorb heterogeneous private data across multiple modalities. Accordingly, we explore the recent advancements and opportunities for such multimodal federated learning in healthcare; we then discuss the key challenges and promising strategies for overcoming these. Abstract: Recent advancements in multimodal machine learning have empowered the development of accurate and robust AI systems in the medical domain, especially within centralized database systems. Simultaneously, Federated Learning (FL) has progressed, providing a decentralized mechanism where data need not be consolidated, thereby enhancing the privacy and security of sensitive healthcare data. The integration of these two concepts supports the ongoing progress of multimodal learning in healthcare while ensuring the security and privacy of patient records within local data-holding agencies. This paper offers a concise overview of the significance of FL in healthcare and outlines the current state-of-the-art approaches to Multimodal Federated Learning (MMFL) within the healthcare domain. It comprehensively examines the existing challenges in the field, shedding light on the limitations of present models. Finally, the paper outlines potential directions for future advancements in the field, aiming to bridge the gap between cutting-edge AI technology and the imperative need for patient data privacy in healthcare applications. Introduction: Artificial intelligence (AI) tools have been transforming several domains (for example, language translation, speech recognition), and in recent years, it has been showing promise in healthcare applications. Most of such demonstrations have been narrowly focused on tasks using a single modality.