Calibrating Biased Distribution in VFM-derived Latent Space via Cross-Domain Geometric Consistency