The robot could learn from these demonstrations to complete the tasks autonomously. For IL algorithms to be useful, however, they must be able to learn how to perform tasks from few demonstrations. A domestic robot wouldn't be very helpful if it required thirty demonstrations before it figured out that you are deliberately washing your purple cravat

By identifying an instrumental variable correlated with the variableX butunrelated tothe confounders, researchers can isolate the exogenous variation inX and estimate acausal relationship betweenX andY.

For our specific algorithm, TD3+BC, given the performance gain over existing state-of-the-art methods is minimal, it would be surprising to see our paper result in significant impact in these contexts. ForCQLwemodify the GitHub defaults for the actor learning rate and use a fixedα rather than the Lagrange variant, matching thehyperparameters definedintheirpaper(whichdiffersfromtheGitHub), aswefound theoriginal hyperparameters performed better. We can also chooseλ by considering the value estimate of the agent-if we see divergence in the value function due to extrapolation error [Fujimoto et al., 2019], then we need to decreaseλ such that the BC term is weightedmorehighly. We use the default hyperparameters in the Fisher-BRC GitHub. Figure 1: Percent difference of performance of offline RL algorithms when adding normalization to state features.

Unfortunately, the main benefit of offline RL, the lack of environment interaction, is also what makes it a challenging task.

Ontheotherhand,asintelligentagents,humansusuallyspendlesstime learning similar tasks and can acquire new skills using existing ones.

Weconsider Diplomacy,a7-player board game designed toaccentuate dilemmas resulting from many-agent interactions.