Protecting multimodal large language models against misleading visualizations

Visualizations play a pivotal role in daily communication in an increasingly data-driven world. Research on multimodal large language models (MLLMs) for automated chart understanding has accelerated massively, with steady improvements on standard benchmarks. However, for MLLMs to be reliable, they must be robust to misleading visualizations, charts that distort the underlying data, leading readers to draw inaccurate conclusions that may support disinformation. Here, we uncover an important vulnerability: MLLM question-answering accuracy on misleading visualizations drops on average to the level of a random baseline. To address this, we introduce the first inference-time methods to improve performance on misleading visualizations, without compromising accuracy on non-misleading ones. The most effective method extracts the underlying data table and uses a text-only LLM to answer the question based on the table. Our findings expose a critical blind spot in current research and establish benchmark results to guide future efforts in reliable MLLMs. Keywords: large language models, chart understanding, visualization In an increasingly data-driven world, visualizations are widely used by scientists, journalists, governments, or companies to efficiently communicate data insights to a broad audience [1]. The correct answer is colored in green, while the wrong answer supported by the misleader is colored in purple. In many cases, visualizations support a message more convincingly than if the underlying data table was shown directly to readers [3].