We find that the sample size has almost no impact on the learning. Notice that the target KL divergence imposes on Ant Maze is higher than the one on Point Maze. "space" to explore around the composite skill prior. As target KL divergence increases, the learned policy will receive less guidance from the prior. The algorithm is not sensitive to this parameter.

We provide an algorithm table that represents HIGL in Algorithm 1. Algorithm 1 Hierarchical reinforcement learning guided by landmarks (HIGL)Input: Goal transition function h, state-goal mapping function ϕ, high-level action frequency m, RND networks θ, θ Initialize empty adjacency matrix M Initialize priority queue Q for n = 1,...,N do Reset the environment and sample the initial state s Sample episode end signal done . Build a graph with the sampled landmarks, a state and a goal. A simulated ball (point mass) starts at the bottom left corner in a " "-shaped maze and aims to reach the top left corner. Its actions correspond to torques applied to joints. This environment has a " "-shaped maze whose size is We define a "success" as being within an L2 distance Each episode is terminated if the agent reaches the goal or after 500 steps.

Deep reinforcement learning (RL) has demonstrated wide success in a variety of sequential decision-making problems, i.e., board games [

Emergent symbolic representations are critical for enabling developmental learning agents to plan and generalize across tasks. In this work, we investigate whether large language models (LLMs) can translate human natural language instructions into the internal symbolic representations that emerge during hierarchical reinforcement learning. We apply a structured evaluation framework to measure the translation performance of commonly seen LLMs -- GPT, Claude, Deepseek and Grok -- across different internal symbolic partitions generated by a hierarchical reinforcement learning algorithm in the Ant Maze and Ant Fall environments. Our findings reveal that although LLMs demonstrate some ability to translate natural language into a symbolic representation of the environment dynamics, their performance is highly sensitive to partition granularity and task complexity. The results expose limitations in current LLMs capacity for representation alignment, highlighting the need for further research on robust alignment between language and internal agent representations.

We find that the sample size has almost no impact on the learning. Notice that the target KL divergence imposes on Ant Maze is higher than the one on Point Maze. "space" to explore around the composite skill prior. As target KL divergence increases, the learned policy will receive less guidance from the prior. The algorithm is not sensitive to this parameter.

We provide an algorithm table that represents HIGL in Algorithm 1. Algorithm 1 Hierarchical reinforcement learning guided by landmarks (HIGL)Input: Goal transition function h, state-goal mapping function ϕ, high-level action frequency m, RND networks θ, θ Initialize empty adjacency matrix M Initialize priority queue Q for n = 1,...,N do Reset the environment and sample the initial state s Sample episode end signal done . Build a graph with the sampled landmarks, a state and a goal. A simulated ball (point mass) starts at the bottom left corner in a " "-shaped maze and aims to reach the top left corner. Its actions correspond to torques applied to joints. This environment has a " "-shaped maze whose size is We define a "success" as being within an L2 distance Each episode is terminated if the agent reaches the goal or after 500 steps.

Deep reinforcement learning (RL) has demonstrated wide success in a variety of sequential decision-making problems, i.e., board games [

Hierarchical reinforcement learning (HRL) learns to make decisions on multiple levels of temporal abstraction. A key challenge in HRL is that the low-level policy changes over time, making it difficult for the high-level policy to generate effective subgoals. To address this issue, the high-level policy must capture a complex subgoal distribution while also accounting for uncertainty in its estimates. We propose an approach that trains a conditional diffusion model regularized by a Gaussian Process (GP) prior to generate a complex variety of subgoals while leveraging principled GP uncertainty quantification. Building on this framework, we develop a strategy that selects subgoals from both the diffusion policy and GP's predictive mean. Our approach outperforms prior HRL methods in both sample efficiency and performance on challenging continuous control benchmarks.

Goal-conditioned hierarchical reinforcement learning (HRL) decomposes complex reaching tasks into a sequence of simple subgoal-conditioned tasks, showing significant promise for addressing long-horizon planning in large-scale environments. This paper bridges the goal-conditioned HRL based on graph-based planning to brain mechanisms, proposing a hippocampus-striatum-like dual-controller hypothesis. Inspired by the brain mechanisms of organisms (i.e., the high-reward preferences observed in hippocampal replay) and instance-based theory, we propose a high-return sampling strategy for constructing memory graphs, improving sample efficiency. Additionally, we derive a model-free lower-level Q-function gradient penalty to resolve the model dependency issues present in prior work, improving the generalization of Lipschitz constraints in applications. Finally, we integrate these two extensions, High-reward Graph and model-free Gradient Penalty (HG2P), into the state-of-the-art framework ACLG, proposing a novel goal-conditioned HRL framework, HG2P+ACLG. Experimentally, the results demonstrate that our method outperforms state-of-the-art goal-conditioned HRL algorithms on a variety of long-horizon navigation tasks and robotic manipulation tasks.

The ability to plan at many different levels of abstraction enables agents to envision the long-term repercussions of their decisions and thus enables sample-efficient learning. This becomes particularly beneficial in complex environments from high-dimensional state space such as pixels, where the goal is distant and the reward sparse. We introduce Forecaster, a deep hierarchical reinforcement learning approach which plans over high-level goals leveraging a temporally abstract world model. Forecaster learns an abstract model of its environment by modelling the transitions dynamics at an abstract level and training a world model on such transition. It then uses this world model to choose optimal high-level goals through a tree-search planning procedure. It additionally trains a low-level policy that learns to reach those goals. Our method not only captures building world models with longer horizons, but also, planning with such models in downstream tasks. We empirically demonstrate Forecaster's potential in both single-task learning and generalization to new tasks in the AntMaze domain.