Geometric Prior-Guided Federated Prompt Calibration