Novel optical imaging techniques, such as hyperspectral imaging (HSI) combined with machine learning-based (ML) analysis, have the potential to revolutionize clinical surgical imaging. However, these novel modalities face a shortage of large-scale, representative clinical data for training ML algorithms, while preclinical animal data is abundantly available through standardized experiments and allows for controlled induction of pathological tissue states, which is not ethically possible in patients. To leverage this situation, we propose a novel concept called "xeno-learning", a cross-species knowledge transfer paradigm inspired by xeno-transplantation, where organs from a donor species are transplanted into a recipient species. Using a total of 11,268 HSI images from humans as well as porcine and rat models, we show that although spectral signatures of organs differ across species, shared pathophysiological mechanisms manifest as comparable relative spectral changes across species. Such changes learnt in one species can thus be transferred to a new species via a novel "physiology-based data augmentation" method, enabling the large-scale secondary use of preclinical animal data for humans. The resulting ethical, monetary, and performance benefits of the proposed knowledge transfer paradigm promise a high impact of the methodology on future developments in the field.

Robust semantic segmentation of intraoperative image data could pave the way for automatic surgical scene understanding and autonomous robotic surgery. Geometric domain shifts, however, although common in real-world open surgeries due to variations in surgical procedures or situs occlusions, remain a topic largely unaddressed in the field. To address this gap in the literature, we (1) present the first analysis of state-of-the-art (SOA) semantic segmentation networks in the presence of geometric out-of-distribution (OOD) data, and (2) address generalizability with a dedicated augmentation technique termed "Organ Transplantation" that we adapted from the general computer vision community. According to a comprehensive validation on six different OOD data sets comprising 600 RGB and hyperspectral imaging (HSI) cubes from 33 pigs semantically annotated with 19 classes, we demonstrate a large performance drop of SOA organ segmentation networks applied to geometric OOD data. Surprisingly, this holds true not only for conventional RGB data (drop of Dice similarity coefficient (DSC) by 46 %) but also for HSI data (drop by 45 %), despite the latter's rich information content per pixel. Using our augmentation scheme improves on the SOA DSC by up to 67 % (RGB) and 90 % (HSI) and renders performance on par with in-distribution performance on real OOD test data. The simplicity and effectiveness of our augmentation scheme makes it a valuable network-independent tool for addressing geometric domain shifts in semantic scene segmentation of intraoperative data. Our code and pre-trained models are available at https://github.com/IMSY-DKFZ/htc.

Synthetic medical image generation has evolved as a key technique for neural network training and validation. A core challenge, however, remains in the domain gap between simulations and real data. While deep learning-based domain transfer using Cycle Generative Adversarial Networks and similar architectures has led to substantial progress in the field, there are use cases in which state-of-the-art approaches still fail to generate training images that produce convincing results on relevant downstream tasks. Here, we address this issue with a domain transfer approach based on conditional invertible neural networks (cINNs). As a particular advantage, our method inherently guarantees cycle consistency through its invertible architecture, and network training can efficiently be conducted with maximum likelihood training. To showcase our method's generic applicability, we apply it to two spectral imaging modalities at different scales, namely hyperspectral imaging (pixel-level) and photoacoustic tomography (image-level). According to comprehensive experiments, our method enables the generation of realistic spectral data and outperforms the state of the art on two downstream classification tasks (binary and multi-class).

Minimal invasive procedures are often preferred over open surgeries because they result in smaller scars, fewer complications and a quicker recovery of the patients. However, they come at the cost of reduced mobility and perception limitations of the surgeon. In many laparoscopic surgeries, for example, it is necessary to stop the blood flow to a specific organ or tissue region by clamping the arteries responsible for blood supply. This procedure, commonly referred to as ischemia induction, prevents excessive bleeding of patients [Tho+07] and is performed in various procedures, including partial nephrectomy, organ transplantation and anastomosis. After clamping the main arteries, it is highly challenging to assess the perfusion state of the tissue solely based on the available RGB video stream. This holds especially true when selective clamping of a segmental artery is performed, in which ischemia is induced only in the cancerous part of the kidney during partial nephrectomy [McC+14; Bor+13] Traditional approaches to improving surgical vision involve fusing preoperatively acquired images with the situs [MH+18]. Such "offline" methods, however, cannot react on dynamics. The most common approach to ensure correct clamping is based on indocyanine green (ICG) fluorescence: after ICG is injected into the blood stream, it binds to the plasma. The bound ICG travels through the blood stream and accumulates in the internal organs, especially in the kidney and liver, within a minute [Tob+12; Gan+16].