Freehand 3D Ultrasound Imaging: Sim-in-the-Loop Probe Pose Optimization via Visual Servoing

Abstract--Freehand 3D ultrasound (US) imaging using conventional 2D probes offers flexibility and accessibility for diverse clinical applications but faces challenges in accurate probe pose estimation. Traditional methods depend on costly tracking systems, while neural network-based methods struggle with image noise and error accumulation, compromising reconstruction precision. We propose a cost-effective and versatile solution that leverages lightweight cameras and visual servoing in simulated environments for precise 3D US imaging. T o counter occlusions and lighting issues, we introduce an image restoration method that reconstructs occluded regions by matching surrounding texture patterns. For pose estimation, we develop a simulation-in-the-loop approach, which replicates the system setup in simulation and iteratively minimizes pose errors between simulated and real-world observations. V alidations on a soft vascular phantom, a 3D-printed conical model, and a human arm demonstrate the robustness and accuracy of our approach, with Hausdorff distances to the reference reconstructions of 0.359 mm, 1.171 mm, and 0.858 mm, respectively. These results confirm the method's potential for reliable freehand 3D US reconstruction. Project resources are available at https://github.com/Y EDICAL ultrasound (US) is widely used in modern clinical practice due to its low cost, real-time imaging, and lack of ionizing radiation. It serves as a first-line tool in various applications, including obstetrics and emergency medicine. This study was partly supported by the Multiscale Medical Robotics Centre, AIR@InnoHK and SINO-German Mobility Project under Grant M0221. Y ameng Zhang is with the Department of Mechanical Engineering, The University of Hong Kong, Hong Kong SAR, China, and also with the Department of Electronic Engineering, The Chinese University of Hong Kong (CUHK), Hong Kong SAR, China (e-mail: zhangyameng@link.cuhk.edu.hk).