Programming by Demonstration with Situated Semantic Parsing

Programming by Demonstration (PbD) is an approach to programming robots by demonstrating the desired behavior. Speech is a natural, hands-free way to augment demonstrations with control commands that guide the PbD process. However, existing speech interfaces for PbD systems rely on ad-hoc, predefined command sets that are rigid and require user training. Instead, we aim to develop flexible speech interfaces to accommodate user variations and ambiguous utterances. To that end, we propose to use a situated semantic parser that jointly reasons about the user's speech and the robot's state to resolve ambiguities. In this paper, we describe this approach and compare its utility to a rigid speech command interface.