SIRL: Similarity-based Implicit Representation Learning