The key challenge in making this connection is grounding the skills, so that each skill corresponds to a specific goal-conditioned policy. We start by recalling the definition of the discounted state occupancymeasure(Eq.3): p(st+=sg)=(1 γ) X On the second line, we havechanged the bounds of the summation to start at 0, and changed the terms inside the summation accordingly. On the third line, we applied linearity of expectation to movethesummation insidetheexpectation. Onthefourthline,weappliedlinearity ofexpectation again to move the term fort = 0 inside the expectation. Finally, we substituted the definition of rg(s,a)toobtainthedesiredresult. This result means that we are doing policy improvement with approximate Q-values.

We usethisideatoreinterpret aprior RLmethod asperforming contrastivelearning, and then use the idea to propose a much simpler method that achieves similar performance. Across arange ofgoal-conditioned RLtasks, wedemonstrate that contrastive RL methods achieve higher success rates than prior non-contrastive methods, including in the offline RL setting.

In reinforcement learning (RL), it is easier to solve a task if given a good representation.

Self-supervised learning has emerged as a pivotal ingredient behind recent scale-driven breakthroughs, where large unlabeled datasets are used to learn powerful representations. However, in the context of reinforcement learning (RL), self-supervision plays a fundamentally different role. Rather than learning from a static dataset, self-supervised RL focuses on learning optimal control-policies through unlabeled sequential interactions with the environment--without relying on manual reward design or human annotation. Such a learning paradigm can enable scalable robot learning systems that autonomously acquire a broad repertoire of behaviors, generalize across tasks, and adapt to new environments with minimal human intervention [1, 2, 3, 4]. However, achieving this level of autonomy is challenging due to the inherent difficulties of exploration, sparse rewards, and the need for learning robust representations from high-dimensional sensory inputs. Goal-Conditioned Reinforcement Learning (GCRL) provides a natural framework for this paradigm, as it enables agents to learn to reach states sampled from a goal distribution and can be formulated without requiring externally provided rewards or expert supervision. Recent work [5] has explored the use of contrastive representation learning for GCRL-- an approach commonly referred to as Contrastive RL (CRL). This class of methods learns a goal-conditioned Q-function by aligning reparXiv:2507.16139v1

In goal-reaching reinforcement learning (RL), the optimal value function has a particular geometry, called quasimetric structure. This paper introduces Quasimetric Reinforcement Learning (QRL), a new RL method that utilizes quasimetric models to learn optimal value functions. Distinct from prior approaches, the QRL objective is specifically designed for quasimetrics, and provides strong theoretical recovery guarantees. Empirically, we conduct thorough analyses on a discretized MountainCar environment, identifying properties of QRL and its advantages over alternatives. On offline and online goal-reaching benchmarks, QRL also demonstrates improved sample efficiency and performance, across both state-based and image-based observations.

In the same way that the computer vision (CV) and natural language processing (NLP) communities have developed self-supervised methods, reinforcement learning (RL) can be cast as a self-supervised problem: learning to reach any goal, without requiring human-specified rewards or labels. However, actually building a self-supervised foundation for RL faces some important challenges. Building on prior contrastive approaches to this RL problem, we conduct careful ablation experiments and discover that a shallow and wide architecture, combined with careful weight initialization and data augmentation, can significantly boost the performance of these contrastive RL approaches on challenging simulated benchmarks. Additionally, we demonstrate that, with these design decisions, contrastive approaches can solve real-world robotic manipulation tasks, with tasks being specified by a single goal image provided after training.

In reinforcement learning (RL), it is easier to solve a task if given a good representation. While deep RL should automatically acquire such good representations, prior work often finds that learning representations in an end-to-end fashion is unstable and instead equip RL algorithms with additional representation learning parts (e.g., auxiliary losses, data augmentation). How can we design RL algorithms that directly acquire good representations? In this paper, instead of adding representation learning parts to an existing RL algorithm, we show (contrastive) representation learning methods can be cast as RL algorithms in their own right. To do this, we build upon prior work and apply contrastive representation learning to action-labeled trajectories, in such a way that the (inner product of) learned representations exactly corresponds to a goal-conditioned value function. We use this idea to reinterpret a prior RL method as performing contrastive learning, and then use the idea to propose a much simpler method that achieves similar performance. Across a range of goal-conditioned RL tasks, we demonstrate that contrastive RL methods achieve higher success rates than prior non-contrastive methods, including in the offline RL setting. We also show that contrastive RL outperforms prior methods on image-based tasks, without using data augmentation or auxiliary objectives.