Reward-Machine-Guided, Self-Paced Reinforcement Learning

We hypothesize that taking advantage of prior knowledge about the underlying Figure 1: Workflow diagram for an existing self-paced RL task structure can improve the effectiveness approach, and two methods that we propose: intermediate of self-paced RL. We develop a self-paced RL self-paced RL and reward-machine-guided, self-paced RL. algorithm guided by reward machines, i.e., a type of finite-state machine that encodes the underlying task structure. The algorithm integrates reward machines in 1) the update of the policy and value et al. [2017] focus on automating the process of curriculum functions obtained by any RL algorithm of choice, generation. Klink et al. [2020a] adopt self-paced learning and 2) the update of the automated curriculum that [Kumar et al., 2010] in RL by developing an algorithm that generates context distributions. Our empirical results creates a sequence of probability distributions over contexts evidence that the proposed algorithm achieves [Hallak et al., 2015]. The dynamics, the reward function, optimal behavior reliably even in cases in which and the initial state distribution of an environment change existing baselines cannot make any meaningful with respect to the context. Given a target context distribution, progress. It also decreases the curriculum length a self-paced RL algorithm iteratively generates context and reduces the variance in the curriculum generation distributions that maximizes the expected discounted return, process by up to one-fourth and four orders of regularized by the Kullback-Leibler (KL) divergence from magnitude, respectively.