Learning Invariant Representations using Inverse Contrastive Loss