ContextualLVLM-Agent: A Holistic Framework for Multi-Turn Visually-Grounded Dialogue and Complex Instruction Following

--Despite significant advancements in Large Language Models (LLMs) and Large Vision-Language Models (L VLMs), current models still face substantial challenges in handling complex, multi-turn, and visually-grounded tasks that demand deep reasoning, sustained contextual understanding, entity tracking, and multi-step instruction following. Existing benchmarks often fall short in capturing the dynamism and intricacies of real-world multi-modal interactions, leading to issues such as context loss and visual hallucinations. T o address these limitations, we introduce MMDR-Bench (Multi-Modal Dialogue Reasoning Benchmark), a novel dataset comprising 300 meticulously designed complex multi-turn dialogue scenarios, each averaging 5-7 turns and evaluated across six core dimensions including visual entity tracking and reasoning depth. Our extensive experiments on MMDR-Bench demonstrate that CoL VLM Agent consistently achieves superior performance, attaining an average human evaluation score of 4.03, notably surpassing state-of-the-art commercial models like GPT -4o (3.92) and Gemini 1.5 Pro (3.85). The framework exhibits significant advantages in reasoning depth, instruction adherence, and error suppression, and maintains robust performance over extended dialogue turns, validating the effectiveness of its modular design and iterative approach for complex multi-modal interactions. The rapid advancements in Large Language Models (LLMs) [1], [2] and Large Vision-Language Models (L VLMs) [3] have revolutionized our ability to understand and generate text and images, respectively. These models have demonstrated remarkable capabilities across a wide spectrum of tasks, from natural language understanding to complex image captioning and visual question answering. As these models become increasingly sophisticated, there is a growing demand for intelligent systems that can engage in more dynamic, nuanced, and context-aware interactions, mirroring human cognitive processes in real-world scenarios. This necessitates models that can not only comprehend textual instructions but also seamlessly integrate visual information, perform deep reasoning within evolving dialogue contexts, track entities, understand spatial relationships, and ultimately execute multi-step complex instructions [4], [5].