A Geometric Perspective on Self-Supervised Policy Adaptation