Inverse Dynamics Pretraining Learns Good Representations for Multitask Imitation

The goal is to use the pretraining corpus to learn a low dimensional representation of the high dimensional (e.g., visual) observation space which can be transferred to a novel context for finetuning on a limited dataset of demonstrations. Among a variety of possible pretraining objectives, we argue that inverse dynamics modeling - i.e., predicting an action given the observations appearing before and after it in the demonstration - is well-suited to this setting.