Dexterous robot hand teleoperation allows for long-range transfer of human manipulation expertise, and could simultaneously provide a way for humans to teach these skills to robots. However, current methods struggle to reproduce the functional workspace of the human hand, often limiting them to simple grasping tasks. We present a novel method for finger-gaited manipulation with multi-fingered robot hands. Our method provides the operator enhanced flexibility in making contacts by expanding the reachable workspace of the robot hand through residual Gaussian Process learning. We also assist the operator in maintaining stable contacts with the object by allowing them to constrain fingertips of the hand to move in concert. Extensive quantitative evaluations show that our method significantly increases the reachable workspace of the robot hand and enables the completion of novel dexterous finger gaiting tasks. Project website: http://respilot-hri.github.io

Abstract--We build a system that enables any human to control a robot hand and arm, simply by demonstrating motions with their own hand. The robot observes the human operator via a single RGB camera and imitates their actions in real-time. Human hands and robot hands differ in shape, size, and joint structure, and performing this translation from a single uncalibrated camera is a highly underconstrained problem. Moreover, the retargeted trajectories must effectively execute tasks on a physical robot, which requires them to be temporally smooth and free of self-collisions. Our key insight is that while paired humanrobot correspondence data is expensive to collect, the internet contains a massive corpus of rich and diverse human hand videos. Our system leverages passive data from the internet to hand-arm trajectory that is smooth, swift, safe, and semantically enable robotic real-time imitation in-the-wild. We demonstrate that it does not require any special gloves, mocap markers or even camera enables previously untrained people to teleoperate a robot on calibration and works from a single RGB camera. Despite recent advancements, building an easy-to-use, performant and low-cost teleoperation system for Mimicking human behavior with robots has been a central high-dimensional dexterous manipulation has remained elusive. This paradigm, Handa et al. recently proposed DexPilot [19], a low-cost known as teleoperation, has successfully been used to enable system for vision-based teleoperation that is free of markers robots to perform tasks that were unsafe or impossible for or hand-held devices. It lowers the cost and usability barrier, humans to perform, such as handling nuclear materials [37] but relies on a custom setup with multiple calibrated depth or deactivating explosives [16]. Teleoperation has also been cameras, and uses neural networks trained on images collected used to enable the robotic automation of tasks that are easy for in this controlled environment, which limits its use to a specific humans to demonstrate but difficult to program. This means our system that the robot overfits to and repeats verbatim for months or should be low-cost, work for any untrained operator, in years thereafter.