Basis for Intentions: Efficient Inverse Reinforcement Learning using Past Experience

This paper addresses the problem of inverse reinforcement learning (IRL) - inferring the reward function of an agent from observing its behavior. IRL can provide a generalizable and compact representation for apprenticeship learning, and enable accurately inferring the preferences of a human in order to assist them. However, effective IRL is challenging, because many reward functions can be compatible with an observed behavior. We focus on how prior reinforcement learning (RL) experience can be leveraged to make learning these preferences faster and more efficient. We propose the IRL algorithm BASIS (Behavior Acquisition through Successor-feature Intention inference from Samples), which leverages multi-task RL pre-training and successor features to allow an agent to build a strong basis for intentions that spans the space of possible goals in a given domain. When exposed to just a few expert demonstrations optimizing a novel goal, the agent uses its basis to quickly and effectively infer the reward function. Our experiments reveal that our method is highly effective at inferring and optimizing demonstrated reward functions, accurately inferring reward functions from less than 100 trajectories. Inverse reinforcement learning (IRL) seeks to identify a reward function under which observed behavior of an expert is optimal. Once an agent has effectively inferred the reward function, it can then use standard (forward) RL to optimize it, and thus acquire not only useful skills by observing demonstrations, but also a reward function as an explanation for the demonstrator's behavior. By inferring the underlying goal being pursued by the demonstrator, the agent is more likely to be able to generalize to a new scenario in which it must optimize that goal, versus an agent which merely imitates the demonstrated actions. IRL has already proven useful in applications including autonomous driving, where learned models capture the behavior of nearby drivers and pedestrians (Huang et al., 2021; Kim & Pineau, 2016), and is a key component in enabling assistive technologies where a helper agent must infer the goals of the human it is assisting (Hadfield-Menell et al., 2016).