Exploiting Points and Lines in Regression Forests for RGB-D Camera Relocalization

Camera relocalization plays a vital role in many computer vision, robotics, augmented reality (VR) and virtual reality (AR) applications. In the real world, camera relocalization has empowered the recent consumer robotics products such as Dyson 360 Eye and iRobot Roomba 980 to know where they have previously visited [1]. In AR/VR products such as Hololens and Oculus Rift, camera relocalization helps to correctly overlay visual objects in an image sequence or real world. Scene Coordinate Regression Forests (SCRF) [2] is the pioneer in using machine learning for camera relocalization. In this method, a regression forest is trained to infer an estimate of each pixel's correspondence to 3D points in the world coordinate. Then these correspondences are used to infer the camera pose with a robust optimization scheme. Since then, various machine learning based methods, mainly random forests based [3], [4], [5], [6], [7], [8], [9] and deep learning based methods [10], [11], [12], [13], [14], [15] have been proposed to accelerate the progress of camera relocalization, in parallel with the traditional but still active featurebased methods [16], [17] and key-frame based methods [18], [19]. In these random forests based methods, either RGB-D/RGB pixel comparison features [2], [3], [5], [7], or the sparse features such as SIFT [8] are employed, without considering the spatial structure.