Geometry-Aware Adaptation for Pretrained Models