On the Utility of Model Learning in HRI

Abstract--Fundamental to robotics is the debate between model-based and model-free learning: should the robot build an explicit model of the world, or learn a policy directly? In the context of HRI, part of the world to be modeled is the human. One option is for the robot to treat the human as a black box and learn a policy for how they act directly. But it can also model the human as an agent, and rely on a "theory of mind" to guide or bias the learning (grey box). We contribute a characterization of the performance of these methods under the optimistic case of having an ideal theory of mind, as well as under different scenarios in which the assumptions behind the robot's theory of mind for the human are wrong, as they inevitably will be in practice. We find that there is a significant sample complexity advantage to theory of mind methods and that they are more robust to covariate shift, but that when enough interaction data is available, black box approaches eventually dominate. An age-old debate that still animates the halls of computer science, robotics, neuroscience, and psychology departments alike is that between model-based and model-free (reinforcement) learning. Model-based methods work by building a model of the world - the dynamics that tells an agent how the world state will change as a consequence of its actions - and optimizing a cost or reward function under the learned model. In contrast, model-free methods never attempt to explicitly learn how the world works. Instead, the agent learns a policy directly from acting in the world and learning from what works and what does not. Model-free methods are appealing because the agent implicitly learns what it needs to know about the world, and only what it needs. Model-based methods are appealing because knowing how the world works might enable the agent to generalize beyond its experience, and possibly be able to explain why a decision is the best one.