Fast and Robust Visuomotor Riemannian Flow Matching Policy