Evaluating Pointing Gestures for Target Selection in Human-Robot Collaboration

-- Pointing gestures are a common interaction method used in Human-Robot Collaboration for various tasks, ranging from selecting targets to guiding industrial processes. This study introduces a method for localizing pointed targets within a planar workspace. The approach employs pose estimation, and a simple geometric model based on shoulder-wrist extension to extract gesturing data from an RGB-D stream. The study proposes a rigorous methodology and comprehensive analysis for evaluating pointing gestures and target selection in typical robotic tasks. In addition to evaluating tool accuracy, the tool is integrated into a proof-of-concept robotic system, which includes object detection, speech transcription, and speech synthesis to demonstrate the integration of multiple modalities in a collaborative application. Finally, a discussion over tool limitations and performance is provided to understand its role in multimodal robotic systems. Deictic gestures are a natural way to interact with the world to identify objects of interest [1]. In collaborative robotic systems, pointing gestures can be used as a powerful tool to perform decision-making, such as target selection. Using gestures has its advantage especially in industrial environments, where interpreting speech commands can be difficult due to noise. Localizing pointing gestures creates a new layer of information that can be used as a basis for gestural algorithms in collaborative systems.