DreamGrasp: Zero-Shot 3D Multi-Object Reconstruction from Partial-View Images for Robotic Manipulation

Robot tasks such as target-driven manipulation and 3D spatial reasoning - like collision avoidance and path planning - require the ability to reconstruct the 3D geometry of objects and identify object instances from visual observations. While many existing methods rely on accurate depth [1, 2] or all-around-view RGB images [3, 4, 5], we address a more practical setting: reconstruction from only sparse, partial-view RGB images (e.g., as few as two). This direction is crucial for real-world deployment, where depth sensing often fails for transparent or reflective objects, and full 360 scene coverage is rarely available due to occlusions in cluttered environments or the limited workspace of a robot. Recent approaches address this challenge by employing supervised learning on datasets of partial observations paired with complete scenes [6, 7]. However, the performance of these supervised learning approaches is inherently constrained by the diversity of the datasets. This highlights the need for alternative approaches that can generalize more robustly beyond the constraints of curated 3D training data. We draw inspiration from recent findings that large-scale image generative models - when suitably fine-tuned with datasets of paired camera poses and images - can produce reasonable predictions of unseen parts of a scene (e.g., the backs of objects) given a partial-view image as input [8, 9, 10, 11, 12, 13, 14]. However, prior works demonstrate strong performance only on clean, front-view images of single, unoccluded objects, and often struggle in multi-object scenarios where objects heavily occlude one another - which are prevalent in robotic object manipulation tasks. Our key idea is to first reconstruct a coarse 3D geometry from partial-view images, and then segment multiple objects into individual instances - where we adopt the recent contrastive lifting algoarXiv:2507.05627v1