Shared Manifold Learning Using a Triplet Network for Multiple Sensor Translation and Fusion with Missing Data

Abstract--Heterogeneous data fusion can enhance the robustness and accuracy of an algorithm on a given task. However, due to the difference in various modalities, aligning the sensors and embedding their information into discriminative and compact representations is challenging. In this paper, we propose a Contrastive learning based MultiModal Alignment Network (CoMMANet) to align data from different sensors into a shared and discriminative manifold where class information is preserved. The proposed architecture uses a multimodal triplet autoencoder to cluster the latent space in such a way that samples of the same classes from each heterogeneous modality are mapped close to each other. Since all the modalities exist in a shared manifold, a unified classification framework is proposed. A comparison made with other methods demonstrates the superiority of this method. This method is also called decision fusion. In the context of a neural network, these outstanding results on tasks like land-use and land-cover representations are generated by the convolutional layers classification (LULC) [1] [2], mineral exploration [3] [4] and fused gradually to form a shared representation [5], urban planning [6], biodiversity conservation [7], sentiment layer. In Fusion methods can be classified into two groups: concatenation and alignment-based methods. Personal use of this material is permitted. To increase the interpretability learn spatial information by using a structured morphological of fusion models, Hong et al. [27] proposed a element of predefined size and shape. They proposed a graphbased shared and specific feature learning (S2FL) that is capable of model to couple the dimension reduction and fusion of decomposing data into modality-shared and modality-specific information. However, using this method, the cloud-covered components, which enables a better information blending of regions are not accurately classified because the morphological multiple heterogeneous modalities.