DiffGAP: A Lightweight Diffusion Module in Contrastive Space for Bridging Cross-Model Gap