Regarding state estimation, many works have explored The combination of cameras and inertial measurement units to use multiple vision sensors for better VINS performance (IMUs) have become prevalent in autonomous vehicles and (Leutenegger et al. 2015; Usenko et al. 2016; Paul mobile devices in the recent decade due to their decrease in et al. 2017; Sun et al. 2018; Kuo et al. 2020; Campos cost and complementary sensing nature. A camera provides et al. 2021; Fu et al. 2021). In particular, Leutenegger texture-rich images of 2 degree-of-freedom (DoF) bearing et al. (2015), Usenko et al. (2016) and Fu et al. (2021) observations to environmental features, while a 6-axis IMU have shown that stereo camera or multiple cameras can typically consists of a gyroscope and an accelerometer achieve better pose accuracy or lower the uncertainties which measures high-frequency angular velocity and linear of IMU-Camera calibration. Only a few works recently acceleration, respectively. This has lead to a significant investigate multiple inertial sensor fusion for VINS (Kim progress of developing visual-inertial navigation system et al. 2017; Eckenhoff et al. 2019b; Zhang et al. 2020; (VINS) algorithms focusing on efficient and accurate pose Wu et al. 2023; Faizullin and Ferrer 2023), showing that estimation (Huang 2019). While many works have shown the system robustness and pose accuracy can be improved accurate estimation for the minimal sensing case of a single by fusing additional IMUs. For optimal fusion of multiple camera and IMU (Mourikis and Roumeliotis 2007; Bloesch asynchronous visual and inertial sensors for MVIS, et al. 2015; Forster et al. 2016; Qin et al. 2018; Geneva et al. it is crucial to provide accurate full-parameter calibration 2020), it is known that the inclusion of additional sensors for these sensors, which include: (i) IMU-IMU/camera can provide improved accuracy due to additional information rigid transformation, (ii) IMU-IMU/camera time offset, (iii) and robustness to single sensor failure cases (Paul et al.

Robust multisensor fusion of multi-modal measurements such as IMUs, wheel encoders, cameras, LiDARs, and GPS holds great potential due to its innate ability to improve resilience to sensor failures and measurement outliers, thereby enabling robust autonomy. To the best of our knowledge, this work is among the first to develop a consistent tightly-coupled Multisensor-aided Inertial Navigation System (MINS) that is capable of fusing the most common navigation sensors in an efficient filtering framework, by addressing the particular challenges of computational complexity, sensor asynchronicity, and intra-sensor calibration. In particular, we propose a consistent high-order on-manifold interpolation scheme to enable efficient asynchronous sensor fusion and state management strategy (i.e. dynamic cloning). The proposed dynamic cloning leverages motion-induced information to adaptively select interpolation orders to control computational complexity while minimizing trajectory representation errors. We perform online intrinsic and extrinsic (spatiotemporal) calibration of all onboard sensors to compensate for poor prior calibration and/or degraded calibration varying over time. Additionally, we develop an initialization method with only proprioceptive measurements of IMU and wheel encoders, instead of exteroceptive sensors, which is shown to be less affected by the environment and more robust in highly dynamic scenarios. We extensively validate the proposed MINS in simulations and large-scale challenging real-world datasets, outperforming the existing state-of-the-art methods, in terms of localization accuracy, consistency, and computation efficiency. We have also open-sourced our algorithm, simulator, and evaluation toolbox for the benefit of the community: https://github.com/rpng/mins.

Achieving accurate, efficient, and consistent localization within an a priori environment map remains a fundamental challenge in robotics and computer vision. Conventional map-based keyframe localization often suffers from sub-optimal viewpoints due to limited field of view (FOV), thus degrading its performance. To address this issue, in this paper, we design a real-time tightly-coupled Neural Radiance Fields (NeRF)-aided visual-inertial navigation system (VINS), termed NeRF-VINS. By effectively leveraging NeRF's potential to synthesize novel views, essential for addressing limited viewpoints, the proposed NeRF-VINS optimally fuses IMU and monocular image measurements along with synthetically rendered images within an efficient filter-based framework. This tightly coupled integration enables 3D motion tracking with bounded error. We extensively compare the proposed NeRF-VINS against the state-of-the-art methods that use prior map information, which is shown to achieve superior performance. We also demonstrate the proposed method is able to perform real-time estimation at 15 Hz, on a resource-constrained Jetson AGX Orin embedded platform with impressive accuracy.