SCOOP'D: Learning Mixed-Liquid-Solid Scooping via Sim2Real Generative Policy

Our method, SCOOP'D, is trained entirely in simulation (left) and generalizes to diverse real-world scenarios (middle) and robust conditions (right). The left column shows simulation demonstrations and testing examples. The middle column (I-VI) presents real-world scenes with various objects and environments, where yellow circles denote the target objects. The right column demonstrates the robustness of our learned policy under different disturbances such as human perturbations, lighting changes, and different camera viewpoints. Abstract-- Scooping items with tools such as spoons and ladles is common in daily life, ranging from assistive feeding to retrieving items from environmental disaster sites. However, developing a general and autonomous robotic scooping policy is challenging since it requires reasoning about complex tool-object interactions. Furthermore, scooping often involves manipulating deformable objects, such as granular media or liquids, which is challenging due to their infinite-dimensional configuration spaces and complex dynamics. We propose a method, SCOOP'D, which uses simulation from OmniGibson (built on NVIDIA Omniverse) to collect scooping demonstrations using algorithmic procedures that rely on privileged state information. Then, we use generative policies via diffusion to imitate demonstrations from observational input. We directly apply the learned policy in diverse real-world scenarios, testing its performance on various item quantities, item characteristics, and container types. In zero-shot deployment, our method demonstrates promising results across 465 trials in diverse scenarios, including objects of different difficulty levels that we categorize as "Level 1" and "Level 2." SCOOP'D outperforms all baselines and ablations, suggesting that this is a promising approach to acquiring robotic scooping skills. Project page is at https://scoopdiff.github.io/. Scooping with tools like ladles or spoons is a fundamental skill in tasks ranging from cooking [12], assistive feeding [42], [44] and environmental cleanup [41]. Also, observations are often unreliable due to occlusions and fluid surfaces that cause reflections, refractions, and unstable depth sensing, which makes scene representations noisy and unreliable.