Towards a Dataset for Human Computer Communication via Grounded Language Acquisition

The Natural Language Processing, Artificial Intelligence, and Robotics fields have made significant progress towards developing robust component technologies (speech recognition/synthesis, machine translation, image recognition); advanced inference mechanisms that accommodate uncertainty and noise; and autonomous driving systems that operate seamlessly on our roads. In spite of this, we do not yet know how to talk to the machines we build or have them speak to us in natural language; how to make them smarter via simple, natural language instructions; how to understand what they are about to do; or how to work with them collaboratively towards accomplishing some joint goal. In this paper, we discuss our work towards building a dataset that enables an empirical approach to studying the relation between natural language, actions, and plans; and introduce a problem formulation that allows us to take meaningful steps towards addressing the open problems listed above.