Bridging Spectral-wise and Multi-spectral Depth Estimation via Geometry-guided Contrastive Learning