Active Imitation Learning via Reduction to I.I.D. Active Learning

In standard passive imitation learning, the goal is to learn an expert’s policy by passively observing full execution trajectories of it. Unfortunately, generating such trajectories can require substantial expert effort and be impractical in some cases. In this paper, we consider Active Imitation Learning (AIL) with the goal of reducing this effort by querying the expert about the desired action at individual states, which are selected based on answers to past queries and the learner’s interactions with an environment simulator. Our new approach is based on reducing AIL to i.i.d. active learning, which can leverage progress in the i.i.d. setting. We introduce and analyze reductions for both non-stationary and stationary policies, showing that the label complexity (number of queries) of AIL can be substantially less than passive learning. We also introduce a practical algorithm inspired by the reductions, which is shown to be highly effective in four test domains compared to a number of alternatives.