Behavioral cloning reduces policy learning to supervised learning by training a discriminative model to predict expert actions given observations.

Summary: This paper has a very interesting claim: distributional shift in imitation learning settings is primarily caused by causal misidentification of the features by the learning algorithm. An interesting example is that of a self-driving car policy trained on a dataset of paired image-control datapoints collected by an expert human driving the car. If the images contain the turn signal on the dashboard then the supervised learner learns to have very good predictive power by indexing on that feature in the image. Clearly that does not generalize during test time. While this is a pathological example, such behavior is present in most settings where usually the state is blown-up by appending past states and actions.

A novel imitation learning algorithm is introduced by applying a game-theoretic notion of correlated equilibrium to the generative adversarial imitation learning. This imitation learning algorithm is equipped with queues of discriminators and agents, in contrast with the classical approach, where there are single discriminator and single agent. The achievement of a correlated equilibrium is due to a mediating neural architecture, which augments the observations that are being seen by queues of discriminators and agents. At every step of the training, the mediator network computes feedback using the rewards of discriminators and agents, to augment the next observations accordingly. By interacting in the game, it steers the training dynamic towards more suitable regions. The resulting imitation learning provides three important benefits. First, it makes adaptability and transferability of the learned model to new environments straightforward. Second, it is suitable for imitating a mixture of state-action trajectories. Third, it avoids the difficulties of non-convex optimization faced by the discriminator in the generative adversarial type architectures.