3D Affordance Keypoint Detection for Robotic Manipulation

Abstract-- This paper presents a novel approach for affordance-informed robotic manipulation by introducing 3D keypoints to enhance the understanding of object parts' functionality. The proposed approach provides direct information about what the potential use of objects is, as well as guidance on where and how a manipulator should engage, whereas conventional methods treat affordance detection as a semantic segmentation task, focusing solely on answering the what question. T o address this gap, we propose a Fusion-based Affordance Keypoint Network (FAKP-Net) by introducing 3D keypoint quadruplet that harnesses the synergistic potential of RGB and Depth image to provide information on execution position, direction, and extent. Benchmark testing demonstrates that FAKP-Net outperforms existing models by significant margins in affordance segmentation task and keypoint detection task. I. INTRODUCTION Autonomous robotic manipulation requires robots to understand the various potential functions of objects and this understanding is referred to as "affordance" [1]. Unlike other properties such as object pose that solely describes the object itself, affordances consider the functional interactions between an object's parts and humans or robots [2]. According to recent studies, the affordance detection for object parts has been approached as a semantic segmentation problem [3], [4], [5], [6], [7], [8], [9], [10] where affordances are predicted by grouping pixels with similar functionality into a single category.