Multimodal Alignment of Histopathological Images Using Cell Segmentation and Point Set Matching for Integrative Cancer Analysis

Abstract: Histopathological imaging is vital for cancer research and clinical practice, with multiplexed Immunofluorescence (MxIF) and Hematoxylin and Eosin (H&E) providing complementary insights. However, aligning different stains at the cell level remains a challenge due to modality differences. In this paper, we present a novel framework for multimodal image alignment using cell segmentation outcomes. By treating cells as point sets, we apply Coherent Point Drift (CPD) for initial alignment and refine it with Graph Matching (GM). Evaluated on ovarian cancer tissue microarrays (TMAs), our method achieves high alignment accuracy, enabling integration of cell-level features across modalities and generating virtual H&E images from MxIF data for enhanced clinical interpretation. Keywords: Histopathology alignment, Histopathology registration, Bioimage analysis Introduction: As an important approach to reveal cell level details in cancer, histopathological images have been widely used in both clinic practice for diagnostic decision making and treatment follow up. Following different staining protocols, each modality of histopathology has its unique strength in highlighting specific aspects within tumor immune microenvironment (TIME). Among which, multiplexed Immunofluorescence (MxIF) images provide refined immune cell phenotyping, making it a favorable research tool for revealing cell behaviors in TIME. However, this imaging technique now is mainly used for research purposes due to the low reliability of marker signals caused by complex cyclic staining processes. On the other hand, H&E (Hematoxylin and Eosin) staining plays an irreplaceable role in providing standard clinical references by revealing cell morphology and texture patterns.