Utilizing Vision-Language Models as Action Models for Intent Recognition and Assistance

Utilizing Vision-Language Models as Action Models for Intent Recognition and Assistance (Extended Abstract) Cesar Alan Contreras 1, Manolis Chiou 2, Alireza Rastegarpanah 3, Michal Szulik 4, Rustam Stolkin 1 1 School of Metallurgy & Materials, University of Birmingham, Birmingham B15 2SE, United Kingdom 2 School of Electronic Engineering and Computer Science, Queen Mary University of London, London E14 4NS, United Kingdom 3 School of Computer Science and Digital Technologies, Aston University, Birmingham B4 7ET, United Kingdom 4 United Kingdom National Nuclear Laboratory Ltd., Warrington W A3 6AE, United Kingdom Abstract --Human-robot collaboration requires robots to quickly infer user intent, provide transparent reasoning, and assist users in achieving their goals. Our recent work introduced GUIDER, our framework for inferring navigation and manipulation intents. We propose augmenting GUIDER with a vision-language model (VLM) and a text-only language model (LLM) to form a semantic prior that filters objects and locations based on the mission prompt. A vision pipeline (YOLO for object detection and the Segment Anything Model for instance segmentation) feeds candidate object crops into the VLM, which scores their relevance given an operator prompt; in addition, the list of detected object labels is ranked by a text-only LLM. Once the combined belief exceeds a threshold, autonomy changes occur, enabling the robot to navigate to the desired area and retrieve the desired object, while adapting to any changes in the operator's intent.