UV-Based 3D Hand-Object Reconstruction with Grasp Optimization

This set-up lends itself well to AR/VR settings where the hand interacts with a predefined object, perhaps with markers to facilitate the object pose estimation. Such a setting is common, although the majority of previous works [31, 33, 34] consider 3D point clouds as input, while we handle the more difficult case of monocular RGB inputs. Additionally, the previous works [12, 13, 31, 33, 34, 60] are singularly focused on reconstructing feasible hand-object interactions. They aim to produce hand meshes with minimal penetration to the 3D object without regard for the accuracy of the 3D hand pose. We take on the additional challenge of balancing realistic hand-object interactions with accurate 3D hand poses. Representation-wise, previous hand-object 3D reconstruction works [2, 35, 61, 72] predominantly with the MANO model [55].