Bigger picture Building efficient multimodal AI algorithms from biomedical data, including medical imaging, electronic health records, and personalized sensors would have immense utility for a broad range of health applications. An increasingly common approach towards building such algorithms is a centralized framework, where all the training databases are held together for training models. However, especially in healthcare, due to variety of reasons, including privacy, security, and ethical reasons, gathering datasets from different hospitals, and health centers is not feasible. Federated learning has emerged as an elegant alternative to centralized machine learning where models are trained separately and collaboratively without a centralized database. However, must of the federated learning models have shown their efficacy in the unimodal system, and there is an increasing need for advanced FL algorithms to allow the training of larger and capable model that can absorb heterogeneous private data across multiple modalities. Accordingly, we explore the recent advancements and opportunities for such multimodal federated learning in healthcare; we then discuss the key challenges and promising strategies for overcoming these. Abstract: Recent advancements in multimodal machine learning have empowered the development of accurate and robust AI systems in the medical domain, especially within centralized database systems. Simultaneously, Federated Learning (FL) has progressed, providing a decentralized mechanism where data need not be consolidated, thereby enhancing the privacy and security of sensitive healthcare data. The integration of these two concepts supports the ongoing progress of multimodal learning in healthcare while ensuring the security and privacy of patient records within local data-holding agencies. This paper offers a concise overview of the significance of FL in healthcare and outlines the current state-of-the-art approaches to Multimodal Federated Learning (MMFL) within the healthcare domain. It comprehensively examines the existing challenges in the field, shedding light on the limitations of present models. Finally, the paper outlines potential directions for future advancements in the field, aiming to bridge the gap between cutting-edge AI technology and the imperative need for patient data privacy in healthcare applications. Introduction: Artificial intelligence (AI) tools have been transforming several domains (for example, language translation, speech recognition), and in recent years, it has been showing promise in healthcare applications. Most of such demonstrations have been narrowly focused on tasks using a single modality.

Formalizing surgical activities as triplets of the used instruments, actions performed, and target anatomies is becoming a gold standard approach for surgical activity modeling. The benefit is that this formalization helps to obtain a more detailed understanding of tool-tissue interaction which can be used to develop better Artificial Intelligence assistance for image-guided surgery. Earlier efforts and the CholecTriplet challenge introduced in 2021 have put together techniques aimed at recognizing these triplets from surgical footage. Estimating also the spatial locations of the triplets would offer a more precise intraoperative context-aware decision support for computer-assisted intervention. This paper presents the CholecTriplet2022 challenge, which extends surgical action triplet modeling from recognition to detection. It includes weakly-supervised bounding box localization of every visible surgical instrument (or tool), as the key actors, and the modeling of each tool-activity in the form of triplet. The paper describes a baseline method and 10 new deep learning algorithms presented at the challenge to solve the task. It also provides thorough methodological comparisons of the methods, an in-depth analysis of the obtained results across multiple metrics, visual and procedural challenges; their significance, and useful insights for future research directions and applications in surgery.