Performance-Guided Refinement for Visual Aerial Navigation using Editable Gaussian Splatting in FalconGym 2.0

Next, we further improve the architecture of [3] by removing IMU inputs and instead feeding a short history of past controls to the controller (blue box in Figure 2), which provides implicit temporal context. Next, the controller training follows a similar imitation learning procedure as in [3]: we first implement a state-based expert that flies through different tracks in simulation; at each timestep, we render the onboard RGB image and record the state-based controller's expert action. The RGB image is passed through the trained U-Net to obtain a binary mask, and we form supervised pairs where the masked image coupled with the past control actions are used to predict the current action to train the controller. Thanks to the Edit API, now we can synthesize essentially arbitrarily many tracks in FalconGym 2.0 to train both perception and controller without additional per-track real-world effort required by [1], [3], [5]. To sample efficiently, our unique design choice is to train on two-gate tracks. Intuitively, the initial state together with two successive gates spans the local geometric variability of longer courses; a controller that performs well on such segments could generalize well to multi-gate tracks by invariance and composition, as is empirically confirmed in Section IV. C. Performance-Guided Refinement Training A straightforward method to collect training data for the visual policy would be to uniformly sample the two-gate track space that is dynamically feasible and observable (as defined at the start of this section). However, uniform sampling can be sample-inefficient in a large high-dimensional workspace. With our Edit API, we can steer training data col- lection toward the visual policy's weak spots and iteratively refine to improve the visual policy.