Abstraction is key to scaling up reinforcement learning (RL). However, autonomously learning abstract state and action representations to enable transfer and generalization remains a challenging open problem. This paper presents a novel approach for inventing, representing, and utilizing options, which represent temporally extended behaviors, in continual RL settings. Our approach addresses streams of stochastic problems characterized by long horizons, sparse rewards, and unknown transition and reward functions. Our approach continually learns and maintains an interpretable state abstraction, and uses it to invent high-level options with abstract symbolic representations. These options meet three key desiderata: (1) composability for solving tasks effectively with lookahead planning, (2) reusability across problem instances for minimizing the need for relearning, and (3) mutual independence for reducing interference among options. Our main contributions are approaches for continually learning transferable, generalizable options with symbolic representations, and for integrating search techniques with RL to efficiently plan over these learned options to solve new problems. Empirical results demonstrate that the resulting approach effectively learns and transfers abstract knowledge across problem instances, achieving superior sample efficiency compared to state-of-the-art methods.

While reinforcement learning (RL) has recently shown many remarkable successes, e.g., in playing Atari and Go at a superhuman level [1, 2], its large sample complexity is still a key problem limiting its application in various fields, e.g., robotics. Allowing robots to learn transferable and reusable options [3] (i.e., skills) is a promising approach to alleviate the issue of sample complexity. As such, an important problem is to characterise option policies by abstract options that can be transferred and instantiated across different environments. Figure 1 illustrates this by a motivating example: given option policies for "find a key" and "open a door" in a 2-room setting, the robot encodes the abstract options and grounds them in an previously unseen 3-room environment. Moreover, the robot can form an abstract semi-Markov Decision Process (SMDP) for planning to navigate in the 3-room environment using the options. In this work, we propose abstract option representations that can be: (1) shared across different environments, (2) grounded in unseen environments with a certain precision, and (3) used for planning with near-optimal performance. In the context of RL, however, options are described by policies which are typically not transferable due to new state spaces and transition dynamics. This issue is underlined by the fact that abstract options such as "open a door" can often correspond to different policies in different MDPs. To enable transferable abstract options, we define shared features across different environments (e.g., doors), and abstract options as successor features [4, 5], which can effectively capture the feature Figure 1: Motivating Example expectations of the option trajectories.