GeoGNN: Quantifying and Mitigating Semantic Drift in Text-Attributed Graphs