HapticVLM: VLM-Driven Texture Recognition Aimed at Intelligent Haptic Interaction

-- This paper introduces HapticVLM, a novel mul-timodal system that integrates vision-language reasoning with deep convolutional networks to enable real-time haptic feedback. HapticVLM leverages a ConvNeXt-based material recognition module to generate robust visual embeddings for accurate identification of object materials, while a state-of-the-art Vision-Language Model (Qwen2-VL-2B-Instruct) infers ambient temperature from environmental cues. Experimental evaluations demonstrate an average recognition accuracy of 84.67% across five distinct auditory-tactile patterns and a temperature estimation accuracy of 86.7% based on a tolerance-based evaluation method with an 8 C margin of error across 15 scenarios. Although promising, the current study is limited by the use of a small set of prominent patterns and a modest participant pool. Future work will focus on expanding the range of tactile patterns and increasing user studies to further refine and validate the system's performance. Overall, HapticVLM presents a significant step toward context-aware, multimodal haptic interaction with potential applications in virtual reality, and assistive technologies. I. INTRODUCTION The ability to perceive and distinguish material properties such as texture, temperature, and stiffness is a fundamental aspect of human interaction with the physical world. Human tactile perception integrates visual, auditory, and haptic cues to form a comprehensive understanding of object surfaces, enabling precise material recognition and interaction [1].