LangGrasp: Leveraging Fine-Tuned LLMs for Language Interactive Robot Grasping with Ambiguous Instructions

Abstract-- The existing language-driven grasping methods struggle to fully handle ambiguous instructions containing implicit intents. T o tackle this challenge, we propose LangGrasp, a novel language-interactive robotic grasping framework. The framework integrates fine-tuned large language models (LLMs) to leverage their robust commonsense understanding and environmental perception capabilities, thereby deducing implicit intents from linguistic instructions and clarifying task requirements along with target manipulation objects. Furthermore, our designed point cloud localization module, guided by 2D part segmentation, enables partial point cloud localization in scenes, thereby extending grasping operations from coarse-grained object-level to fine-grained part-level manipulation. Experimental results show that the LangGrasp framework accurately resolves implicit intents in ambiguous instructions, identifying critical operations and target information that are unstated yet essential for task completion. Additionally, it dynamically selects optimal grasping poses by integrating environmental information. This enables high-precision grasping from object-level to part-level manipulation, significantly enhancing the adaptability and task execution efficiency of robots in unstructured environments. I. INTRODUCTION With the increasing deployment of robots in daily environments, natural language based human-robot interaction and manipulation have significantly enhanced efficiency and task executability due to their intuitiveness and flexibility. However, accurately interpreting ambiguous or context dependent linguistic instructions in dynamic, unstructured environments, particularly those containing implicit intents, remains a critical challenge.