A Algorithms. Algorithm 1 Training DHRL 1: sample D

T time-steps, this upper-bound of error rate is also satisfied in all path from s to g . As shown in the table above, the wider the initial distribution, the easier it is for the agent to explore the map. 'fixed initial state distribution' requires less prior information about the state space. Figure 12: Changes in the graph level over the training; DHRL can explore long tasks with'fixed The results are averaged over 4 random seeds and smoothed equally.