Uncertainty-Aware Perception-Based Control for Autonomous Racing

--Autonomous systems operating in unknown environments often rely heavily on visual sensor data, yet making safe and informed control decisions based on these measurements remains a significant challenge. T o facilitate the integration of perception and control in autonomous vehicles, we propose a novel perception-based control approach that incorporates road estimation, quantification of its uncertainty, and uncertainty-aware control based on this estimate. At the core of our method is a parametric road curvature model, optimized using visual measurements of the road through a constrained nonlinear optimization problem. This process ensures adherence to constraints on both model parameters and curvature. By leveraging the Frenet frame formulation, we embed the estimated track curvature into the system dynamics, allowing the controller to explicitly account for perception uncertainty and enhancing robustness to estimation errors based on visual input. We validate our approach in a simulated environment, using a high-fidelity 3D rendering engine, and demonstrate its effectiveness in achieving reliable and uncertainty-aware control for autonomous racing. Robots increasingly rely on visual feedback to navigate and operate in unknown, complex environments. Recent advances demonstrate the potential of visual perception for control tasks [1], [2], enabling robots to make decisions based on high-dimensional sensory inputs. However, safe deployment of autonomous systems requires robust handling of uncertainty throughout the autonomy stack, including perception, planning, and control, to ensure reliability in dynamic and unpredictable settings. Most existing perception-based control methods, however, assume perfect perception and treat its outputs as certain and fully reliable [2], [3]. This decoupled design of the modules can lead to compounding error and cascading failures in safety-critical applications.