As advocated by [39, 38, 18], procedurally generated environments are a promising direction for testing systematic generalization of RL agents.

The checklist follows the references. Please do not modify the questions and only use the provided macros for your answers. Checklist section does not count towards the page limit. Do the main claims made in the abstract and introduction accurately reflect the paper's Did you describe the limitations of your work? Did you discuss any potential negative societal impacts of your work?

In thispaper,weinvestigatewhether onesuchobjective,theWasserstein-1 distance between a policy's state visitation distribution and a target distribution, can be utilized effectivelyforreinforcement learning (RL)tasks.

Through reinforcement learning (RL), we have made massive strides towards solving tasks that haveasingle goal. However,inthe multi-task domain, where an agent needs to reach multiple goals, the choice of training goals can largely affectsampleefficiency. Whenbiologicalagentslearn,thereisoftenanorganized and meaningful order to which learning happens. Inspired by this, we propose setting up an automatic curriculum for goals that the agent needs to solve.

We first raise and tackle a "time synchronization" issue between the agent and the environment in non-stationary reinforcement learning (RL), a crucial factor