Paying Alignment Tax with Contrastive Learning