We provide an algorithm table that represents HIGL in Algorithm 1. Algorithm 1 Hierarchical reinforcement learning guided by landmarks (HIGL)Input: Goal transition function h, state-goal mapping function ϕ, high-level action frequency m, RND networks θ, θ Initialize empty adjacency matrix M Initialize priority queue Q for n = 1,...,N do Reset the environment and sample the initial state s Sample episode end signal done . Build a graph with the sampled landmarks, a state and a goal. A simulated ball (point mass) starts at the bottom left corner in a " "-shaped maze and aims to reach the top left corner. Its actions correspond to torques applied to joints. This environment has a " "-shaped maze whose size is We define a "success" as being within an L2 distance Each episode is terminated if the agent reaches the goal or after 500 steps.

Goal-conditioned hierarchical reinforcement learning (HRL) decomposes complex reaching tasks into a sequence of simple subgoal-conditioned tasks, showing significant promise for addressing long-horizon planning in large-scale environments. This paper bridges the goal-conditioned HRL based on graph-based planning to brain mechanisms, proposing a hippocampus-striatum-like dual-controller hypothesis. Inspired by the brain mechanisms of organisms (i.e., the high-reward preferences observed in hippocampal replay) and instance-based theory, we propose a high-return sampling strategy for constructing memory graphs, improving sample efficiency. Additionally, we derive a model-free lower-level Q-function gradient penalty to resolve the model dependency issues present in prior work, improving the generalization of Lipschitz constraints in applications. Finally, we integrate these two extensions, High-reward Graph and model-free Gradient Penalty (HG2P), into the state-of-the-art framework ACLG, proposing a novel goal-conditioned HRL framework, HG2P+ACLG. Experimentally, the results demonstrate that our method outperforms state-of-the-art goal-conditioned HRL algorithms on a variety of long-horizon navigation tasks and robotic manipulation tasks.