Region-Level Context-Aware Multimodal Understanding

--Despite significant progress, existing research on Multimodal Large Language Models (MLLMs) mainly focuses on general visual understanding, overlooking the ability to integrate textual context associated with objects for a more context-aware multimodal understanding - an ability we refer to as Region-level Context-aware Multimodal Understanding (RCMU). T o address this limitation, we first formulate the RCMU task, which requires models to respond to user instructions by integrating both image content and textual information of regions or objects. T o equip MLLMs with RCMU capabilities, we propose Region-level Context-aware Visual Instruction T uning (RCVIT), which incorporates object information into the model input and enables the model to utilize bounding box coordinates to effectively associate objects' visual content with their textual information. T o address the lack of datasets, we introduce the RCMU dataset, a large-scale visual instruction tuning dataset that covers multiple RCMU tasks. We also propose RC&P-Bench, a comprehensive benchmark that can evaluate the performance of MLLMs in RCMU and multimodal personalized understanding tasks. Additionally, we propose a reference-free evaluation metric to perform a comprehensive and fine-grained evaluation of the region-level context-aware image descriptions. Experimental results indicate that RC-Qwen2-VL models not only achieve outstanding performance on multiple RCMU tasks but also demonstrate successful applications in multimodal RAG and personalized conversation. UL TIMODAL large language models (MLLMs) [1]- [6] expands language models into the multimodal domain by integrating visual encoders, demonstrating outstanding performance across a range of multimodal tasks, such as visual question answering, document understanding, robotic manipulation, among others. However, existing MLLMs primarily focus on general visual understanding, lacking the ability to integrate textual context for a more context-aware multimodal understanding.