BC formulates imitation learning as a supervised learning problem. It needs no in-environment samples, but it suffers from the covariate shift issue [37], often leading totesttimeperformance degradation.

However, there are still quite a few challenges between the traditional RL research and real-worldtasks.

We characterise the phenomenon empirically, verifying that it is not limited to specific algorithm or environment properties.

Offline reinforcement learning (RL) aims to learn policies from pre-existing datasets without further interactions, making it a challenging task. Q-learning algorithms struggle withextrapolation errors inofflinesettings, while supervised learning methods are constrained by model expressiveness.