Neural Fidelity Calibration for Informative Sim-to-Real Adaptation

--Deep reinforcement learning can seamlessly transfer agile locomotion and navigation skills from the simulator to real world. However, bridging the sim-to-real gap with domain randomization or adversarial methods often demands expert physics knowledge to ensure policy robustness. Even so, cutting-edge simulators may fall short of capturing every real-world detail, and the reconstructed environment may introduce errors due to various perception uncertainties. T o address these challenges, we propose Neural Fidelity Calibration (NFC), a novel framework that employs conditional score-based diffusion models to calibrate simulator physical coefficients and residual fidelity domains online during robot execution. Specifically, the residual fidelity reflects the simulation model shift relative to the real-world dynamics and captures the uncertainty of the perceived environment, enabling us to sample realistic environments under the inferred distribution for policy fine-tuning. Our framework is informative and adaptive in three key ways: (a) we fine-tune the pretrained policy only under anomalous scenarios, (b) we build sequential NFC online with the pretrained NFC's proposal prior, reducing the diffusion model's training burden, and (c) when NFC uncertainty is high and may degrade policy improvement, we leverage optimistic exploration to enable "hallucinated" policy optimization. Our framework achieves superior simulator calibration precision compared to state-of-the-art methods across diverse robots with high-dimensional parametric spaces. We study the critical contribution of residual fidelity to policy improvement in simulation and real-world experiments. Notably, our approach demonstrates robust robot navigation under challenging real-world conditions, such as a broken wheel axle on snowy surfaces. Zero-shot sim-to-real reinforcement learning (RL) has empowered agile policy to various robots across soft [74], wheeled [83], aerial [18], and quadruped [45] embodiments. In the context of policy resilience against the real-world diversities, the proximal works in domain randomization (DR) [75] and adversarial training [19] emerge as powerful strategies by artificially introducing noise or attacks into the agent's states. Safety RL, which incorporates safety constraints into the optimization [10], remains tied to DR via exploration of diverse unsafe scenarios. Despite these advancements, expert real-world knowledge is often required to determine domain ranges [48], reconstruct environments [15], or design adversarial scenarios [66]. In theory, one could uniformly sample every domain parameter and environment variation, but this is usually impractical. Y u and L. Liu are with the Luddy School of Informatics, Computing, and Engineering at Indiana University, Bloomington, IN 47408, USA.