Synthesizing Manipulation Sequences for Under-Specified Tasks using Unrolled Markov Random Fields

When interacting with a robot, users often under-specify the tasks to be performed. For example in Figure 5, when asked to pour something, the robot has to infer which cup to pour into and a complete sequence of the navigation and manipulation steps--moving close, grasping, placing, and so on. This sequence not only changes with the task, but also with the perceived state of the environment. As an example, consider the task of a robot fetching a magazine from a desk. The method to perform this task varies depending on several properties of the environment: for example, the robot's relative distance from the magazine, the robot's relative orientation, the thickness of the magazine, and the presence or the absence of other items on top of the magazine. If the magazine is very thin, the robot may have to slide the magazine to the side of the table to pick it up. If there is a mug sitting on top of the magazine, it would have to be moved prior to the magazine being picked up. Thus, especially when the details of the manipulation task are under-specified, the success of executing the task depends on the ability to detect the object and on the ability to sequence the set of primitives (navigation and manipulation controllers) in various ways in response to the environment. In recent years, there have been significant developments in building low-level controllers for robots [34] as well as in perceptual tasks such as object detection from sensor data [20, 11, 35].