Do You Keep an Eye on What I Ask? Mitigating Multimodal Hallucination via Attention-Guided Ensemble Decoding

Recent advancements in Large Vision-Language Models (L VLMs) have significantly expanded their utility in tasks like image captioning and visual question answering. However, they still struggle with object hallucination, where models generate descriptions that inaccurately reflect the visual content by including nonexistent objects or misrepresenting existing ones. While previous methods, such as data augmentation and training-free approaches, strive to tackle this issue, they still encounter scalability challenges and often depend on additional external modules. In this work, we propose Ensemble Decoding (ED), a novel strategy that splits the input image into sub-images and combines logit distributions by assigning weights through the attention map. Furthermore, we introduce ED adaptive plausibility constraint to calibrate logit distribution and FastED, a variant designed for speed-critical applications. Extensive experiments across hallucination benchmarks demonstrate that our proposed method achieves state-of-the-art performance, validating the effectiveness of our approach. Figure 1: Example of object hallucination in L VLM (left). Recent advancements in Large Language Models (LLMs) (Brown, 2020; Touvron et al., 2023a;b; Jiang et al., 2023) have extended their capabilities into the visual domain. In particular, Large Vision-Language Models (L VLMs) (Liu et al., 2023b; Bai et al., 2023; Liu et al., 2024b; Dai et al., 2023; Gong et al., 2023) process visual inputs and generate contextually relevant text, making them effective for tasks such as image captioning and visual question answering. Despite extensive research focusing on optimizing L VLM architectures, training paradigms, and dataset combinations, the persistent issue of object hallucination raises significant concerns about the reliability and applicability of these models (Liu et al., 2023a; Lovenia et al., 2023; Li et al., 2023b; Liu et al., 2024a). Object hallucination occurs when L VLMs inaccurately describe visual content, misrepresenting or introducing nonexistent objects. Object hallucination is especially problematic in applications that demand precise answers, such as autonomous vehicles (Iberraken & Adouane, 2023) and manufacturing systems (Mohammadi Amin et al., 2020).