Large Multimodal Models (LMMs) have demonstrated remarkable capabilities across a wide range of tasks. However, their vulnerability to user gaslighting-the deliberate use of misleading or contradictory inputs-raises critical concerns about their reliability in real-world applications. In this paper, we address the novel and challenging issue of mitigating the negative impact of negation-based gaslighting on LMMs, where deceptive user statements lead to significant drops in model accuracy. Specifically, we introduce GasEraser, a training-free approach that reallocates attention weights from misleading textual tokens to semantically salient visual regions. By suppressing the influence of "attention sink" tokens and enhancing focus on visually grounded cues, GasEraser significantly improves LMM robustness without requiring retraining or additional supervision. Extensive experimental results demonstrate that GasEraser is effective across several leading open-source LMMs on the GaslightingBench. Notably, for LLaVA-v1.5-7B, GasEraser reduces the misguidance rate by 48.2%, demonstrating its potential for more trustworthy LMMs.

"Gaslighting" -- mind manipulating, grossly misleading, downright deceitful -- is Merriam-Webster's word of the year. There wasn't a single event that drove significant spikes in the curiosity, as it usually goes with the chosen word of the year. "It's a word that has risen so quickly in the English language, and especially in the last four years, that it actually came as a surprise to me and to many of us," said Peter Sokolowski, Merriam-Webster's editor at large, in an exclusive interview with The Associated Press ahead of Monday's unveiling. "It was a word looked up frequently every single day of the year," he said. There were deepfakes and the dark web.