Neuro-Symbolic Contrastive Learning for Cross-domain Inference