MDP Abstraction with Successor Features

While reinforcement learning (RL) has recently shown many remarkable successes, e.g., in playing Atari and Go at a superhuman level [1, 2], its large sample complexity is still a key problem limiting its application in various fields, e.g., robotics. Allowing robots to learn transferable and reusable options [3] (i.e., skills) is a promising approach to alleviate the issue of sample complexity. As such, an important problem is to characterise option policies by abstract options that can be transferred and instantiated across different environments. Figure 1 illustrates this by a motivating example: given option policies for "find a key" and "open a door" in a 2-room setting, the robot encodes the abstract options and grounds them in an previously unseen 3-room environment. Moreover, the robot can form an abstract semi-Markov Decision Process (SMDP) for planning to navigate in the 3-room environment using the options. In this work, we propose abstract option representations that can be: (1) shared across different environments, (2) grounded in unseen environments with a certain precision, and (3) used for planning with near-optimal performance. In the context of RL, however, options are described by policies which are typically not transferable due to new state spaces and transition dynamics. This issue is underlined by the fact that abstract options such as "open a door" can often correspond to different policies in different MDPs. To enable transferable abstract options, we define shared features across different environments (e.g., doors), and abstract options as successor features [4, 5], which can effectively capture the feature Figure 1: Motivating Example expectations of the option trajectories.