Attention-Guided Integration of CLIP and SAM for Precise Object Masking in Robotic Manipulation

Attention-Guided Integration of CLIP and SAM for Precise Object Masking in Robotic Manipulation 1 st Muhammad A. Muttaqien Automation Research T eam National Institute of AIST Tokyo, Japan muha.muttaqien@aist.go.jp 2 nd Tomohiro Motoda Automation Research T eam National Institute of AIST Tokyo, Japan tomohiro.motoda@aist.go.jp 3 rd Ryo Hanai Automation Research T eam National Institute of AIST Tokyo, Japan ryo.hanai@aist.go.jp 4 th Domae Y ukiyasu Automation Research T eam National Institute of AIST Tokyo, Japan domae.yukiyasu@aist.go.jp Abstract --This paper introduces a novel pipeline to enhance the precision of object masking for robotic manipulation within the specific domain of masking products in convenience stores. The approach integrates two advanced AI models, CLIP and SAM, focusing on their synergistic combination and the effective use of multimodal data (image and text). Emphasis is placed on utilizing gradient-based attention mechanisms and customized datasets to fine-tune performance. While CLIP, SAM, and Grad-CAM are established components, their integration within this structured pipeline represents a significant contribution to the field. The resulting segmented masks, generated through this combined approach, can be effectively utilized as inputs for robotic systems, enabling more precise and adaptive object manipulation in the context of convenience store products. I NTRODUCTION In recent years, the ability to recognize and manipulate specific objects within well-defined domains, such as products in convenience stores, has become increasingly important in the field of robotic manipulation [1] [2] [3]. As robots are expected to perform more complex tasks in diverse environments, the need for precise object identification and interaction grows, particularly in domains where a high level of accuracy is crucial. For instance, in convenience stores (Figure 1), robots must reliably identify and handle a wide variety of products, each with unique visual characteristics, to automate tasks such as stocking, sorting, and customer assistance.