AnyTeleop: A General Vision-Based Dexterous Robot Arm-Hand Teleoperation System

Figure 1: We present AnyTeleop, a vision-based teleoperation system for a variety of scenarios to solve a wide range of manipulation tasks. AnyTeleop can be used for various robot arms with different robot hands. It also supports teleoperation within different realities, such as IsaacGym (top row), and SAPIEN simulator (middle row), and real world (bottom rows). Abstract--Vision-based teleoperation offers the possibility experiments, AnyTeleop can outperform a previous system that to endow robots with human-level intelligence to physically was designed for a specific robot hardware with a higher interact with the environment, while only requiring low-cost success rate, using the same robot. However, current vision-based teleoperation AnyTeleop leads to better imitation learning performance, systems are designed and engineered towards a particular robot compared with a previous system that is particularly designed model and deploy environment, which scales poorly as the pool for that simulator. of the robot models expands and the variety of the operating environment increases. They can adapt Reality (VR) devices [4, 17, 15], wearable gloves [29, 30], to new robots given only the kinematic model, i.e., URDF handheld controller [47, 48, 20], haptic sensors [12, 23, files. Second, we develop a web-based viewer compatible 52, 55], or motion capture trackers [68]. Fortunately, recent with standard browsers, to achieve simulator-agnostic visualization developments in vision-based teleoperation [2, 24, 16, 26, and enable remote teleoperation across the internet.