Through reinforcement learning (RL), we have made massive strides towards solving tasks that haveasingle goal. However,inthe multi-task domain, where an agent needs to reach multiple goals, the choice of training goals can largely affectsampleefficiency. Whenbiologicalagentslearn,thereisoftenanorganized and meaningful order to which learning happens. Inspired by this, we propose setting up an automatic curriculum for goals that the agent needs to solve.

We first raise and tackle a "time synchronization" issue between the agent and the environment in non-stationary reinforcement learning (RL), a crucial factor

Methods for sequential decision making are often built upon a foundational assumption that the underlying decision process is stationary [Sutton and Barto, 2018]. While this assumption was a cornerstone when laying the theoretical foundations of the field, and while is often reasonable, it isseldom trueinpractice andcanbeunreasonable [Dulac-Arnold etal.,2019].

These selections serve as subgoals that indicate subtasks and guide policy.