I Spy: An Interactive Game-Based Approach to Multimodal Robot Learning

Teaching robots about objects in their environment requires a multimodal correlation of images and linguistic descriptions to build complete feature and object models.  These models can be created manually by collecting images and related keywords and presenting the pairings to robots, but doing so is tedious and unnatural.  This work abstracts the problem of training robots to learn about the world around them by introducing I Spy , an interactive dialogue- and vision-based game in which players place objects in front of a humanoid robot and challenge it to guess which object they have in mind.  The robot gradually learns about the objects and the features which describe them through repeated games, by updating its knowledge with newly captured training images.  This paper details I Spy's learning and gaming processes, describes the approaches taken to extract information from multiple modalities both before and during gameplay, and finally discusses the results of a study designed to evaluate the game's model accuracy over time, its overall performance, and its appeal to human players.