The RoboHelper Project: From Multimodal Corpus to Embodiment on a Robot

In this position paper, we describe the RoboHelper project, its findings and our vision for its future. The long-term goal of RoboHelper is to develop assistive robots for the elderly. The main thesis of our work is that such robots must crucially be able to participate in multimodal dialogues. Contributions of our work to date include the ELDERLY-AT-HOME corpus that we collected and annotated. It consists of 20 task-oriented human-human dialogues between a helper and an elderly person in a fully functional apartment. The unique feature of the corpus is that in addition to video and audio, it includes recordings of physical interaction. Based on this data, we have demonstrated the crucial role that Haptic-Ostensive (H-O) actions play in interpreting language and uncovering a person's intentions. H-O actions manipulate objects, but they also often perform a referring function. Our models were derived on the basis of manually annotated categories. Additional experiments show that we can identify H-O actions using the physical interaction data measured through an unobtrusive sensory glove developed as part of the project. In future work, we will derive models for the robot to decide what to do next (as opposed to interpreting what the interlocutor did); explore other types of physical interactions; and refine preliminary implementations of our models on the Nao robotic platform.