Consistent Zero-Shot Imitation with Contrastive Goal Inference