From Human Hands to Robotic Limbs: A Study in Motor Skill Embodiment for Telemanipulation

Abstract-- This paper presents a teleoperation system for controlling a redundant degree-of-freedom (DOF) robot manipulator using human arm gestures. We propose a GRU-based Variational Autoencoder (VAE) to learn a latent representation of the manipulator's configuration space, capturing its complex joint kinematics. A fully-connected neural network maps human arm configurations into this latent space, allowing the system to mimic and generate corresponding manipulator trajectories in real-time through the VAE decoder. Arrow shows the mapping relationship between the manipulator's For example, an operator can use as agriculture, healthcare medicine, warehousing, and manufacturing. Another have proliferated, such as for image generation and natural approach instead uses an external RGB and RGBD (depth) language generation, for example ChatGPT, Midjourney, camera to estimate the operator's 6-DOF hand pose [4], [5] and Dall-E, to name a few.