RKHS-BA: A Semantic Correspondence-Free Multi-View Registration Framework with Global Tracking

Abstract--This work reports a novel Bundle Adjustment (BA) formulation using a Reproducing Kernel Hilbert Space (RKHS) representation called RKHS-BA. The proposed formulation is correspondence-free, enables the BA to use RGB-D/LiDAR and semantic labels in the optimization directly, and provides a generalization for the photometric loss function commonly used in direct methods. RKHS-BA can incorporate appearance and semantic labels within a continuous spatial-semantic functional representation that does not require optimization via image pyramids. We demonstrate its applications in sliding-window odometry and global LiDAR mapping, which show highly robust performance in extremely challenging scenes and the best tradeoff of generalization and accuracy. I. INTRODUCTION Bundle Adjustment (BA) is widely used in visual perception algorithms such as Simultaneous Localization and Mapping (SLAM) and 3D Reconstruction. It jointly optimizes visual structures and all the camera parameters to construct a spatially-consistent 3D world model [73]. Then, in the optimization step, they minimize reprojected geometric residuals for features observed across multiple frames via multi-view geometry [36, 73]. However, full images need to be stored in the to sparse Hessian structures but relies on correct feature pose graph even in semi-dense approaches [95]. Many works have their illumination invariance presumption is seriously violated been devoted to improving their robustness, such as improving in outdoor situations where complex illumination, changeable frontend feature matching's quality with deep networks [33], weather, and dynamic objects exist. Specifically, we denote various types of visual information, feature association contaminated with outliers is still an open including pixel classes, object instances, intensities, problem [56].