Tosi, Fabio
How NeRFs and 3D Gaussian Splatting are Reshaping SLAM: a Survey
Tosi, Fabio, Zhang, Youmin, Gong, Ziren, Sandström, Erik, Mattoccia, Stefano, Oswald, Martin R., Poggi, Matteo
Over the past two decades, research in the field of Simultaneous Localization and Mapping (SLAM) has undergone a significant evolution, highlighting its critical role in enabling autonomous exploration of unknown environments. This evolution ranges from hand-crafted methods, through the era of deep learning, to more recent developments focused on Neural Radiance Fields (NeRFs) and 3D Gaussian Splatting (3DGS) representations. Recognizing the growing body of research and the absence of a comprehensive survey on the topic, this paper aims to provide the first comprehensive overview of SLAM progress through the lens of the latest advancements in radiance fields. It sheds light on the background, evolutionary path, inherent strengths and limitations, and serves as a fundamental reference to highlight the dynamic progress and specific challenges.
GO-SLAM: Global Optimization for Consistent 3D Instant Reconstruction
Zhang, Youmin, Tosi, Fabio, Mattoccia, Stefano, Poggi, Matteo
Neural implicit representations have recently demonstrated compelling results on dense Simultaneous Localization And Mapping (SLAM) but suffer from the accumulation of errors in camera tracking and distortion in the reconstruction. Purposely, we present GO-SLAM, a deep-learning-based dense visual SLAM framework globally optimizing poses and 3D reconstruction in real-time. Robust pose estimation is at its core, supported by efficient loop closing and online full bundle adjustment, which optimize per frame by utilizing the learned global geometry of the complete history of input frames. Simultaneously, we update the implicit and continuous surface representation on-the-fly to ensure global consistency of 3D reconstruction. Results on various synthetic and real-world datasets demonstrate that GO-SLAM outperforms state-of-the-art approaches at tracking robustness and reconstruction accuracy. Furthermore, GO-SLAM is versatile and can run with monocular, stereo, and RGB-D input.
The Second Monocular Depth Estimation Challenge
Spencer, Jaime, Qian, C. Stella, Trescakova, Michaela, Russell, Chris, Hadfield, Simon, Graf, Erich W., Adams, Wendy J., Schofield, Andrew J., Elder, James, Bowden, Richard, Anwar, Ali, Chen, Hao, Chen, Xiaozhi, Cheng, Kai, Dai, Yuchao, Hoa, Huynh Thai, Hossain, Sadat, Huang, Jianmian, Jing, Mohan, Li, Bo, Li, Chao, Li, Baojun, Liu, Zhiwen, Mattoccia, Stefano, Mercelis, Siegfried, Nam, Myungwoo, Poggi, Matteo, Qi, Xiaohua, Ren, Jiahui, Tang, Yang, Tosi, Fabio, Trinh, Linh, Uddin, S. M. Nadim, Umair, Khan Muhammad, Wang, Kaixuan, Wang, Yufei, Wang, Yixing, Xiang, Mochu, Xu, Guangkai, Yin, Wei, Yu, Jun, Zhang, Qi, Zhao, Chaoqiang
This paper discusses the results for the second edition of the Monocular Depth Estimation Challenge (MDEC). This edition was open to methods using any form of supervision, including fully-supervised, self-supervised, multi-task or proxy depth. The challenge was based around the SYNS-Patches dataset, which features a wide diversity of environments with high-quality dense ground-truth. This includes complex natural environments, e.g. forests or fields, which are greatly underrepresented in current benchmarks. The challenge received eight unique submissions that outperformed the provided SotA baseline on any of the pointcloud- or image-based metrics. The top supervised submission improved relative F-Score by 27.62%, while the top self-supervised improved it by 16.61%. Supervised submissions generally leveraged large collections of datasets to improve data diversity. Self-supervised submissions instead updated the network architecture and pretrained backbones. These results represent a significant progress in the field, while highlighting avenues for future research, such as reducing interpolation artifacts at depth boundaries, improving self-supervised indoor performance and overall natural image accuracy.
NeRF-Supervised Deep Stereo
Tosi, Fabio, Tonioni, Alessio, De Gregorio, Daniele, Poggi, Matteo
We introduce a novel framework for training deep stereo networks effortlessly and without any ground-truth. By leveraging state-of-the-art neural rendering solutions, we generate stereo training data from image sequences collected with a single handheld camera. On top of them, a NeRF-supervised training procedure is carried out, from which we exploit rendered stereo triplets to compensate for occlusions and depth maps as proxy labels. This results in stereo networks capable of predicting sharp and detailed disparity maps. Experimental results show that models trained under this regime yield a 30-40% improvement over existing self-supervised methods on the challenging Middlebury dataset, filling the gap to supervised models and, most times, outperforming them at zero-shot generalization.