Generalization of CNNs on Relational Reasoning with Bar Charts

--This paper presents a systematic study of the generalization of convolutional neural networks (CNNs) and humans on relational reasoning tasks with bar charts. We first revisit previous experiments on graphical perception and update the benchmark performance of CNNs. We then test the generalization performance of CNNs on a classic relational reasoning task: estimating bar length ratios in a bar chart, by progressively perturbing the standard visualizations. We further conduct a user study to compare the performance of CNNs and humans. Our results show that CNNs outperform humans only when the training and test data have the same visual encodings. Otherwise, they may perform worse. We also find that CNNs are sensitive to perturbations in various visual encodings, regardless of their relevance to the target bars. Y et, humans are mainly influenced by bar lengths. Our study suggests that robust relational reasoning with visualizations is challenging for CNNs. Improving CNNs' generalization performance may require training them to better recognize task-related visual properties. EEP neural networks, especially convolutional neural networks (CNNs), are increasingly being adopted in the visualization community for many tasks such as visual question answering [33], [34], automatic visualization design [3], and chart captioning [35], [44]. Despite their widespread use, the crucial question of how well these models generalize to previously unseen visualizations remains less explored. Understanding and enhancing this generalization ability is crucial for the real-world deployment of CNNs. Graphical perception [5] refers to the human ability to decode visually encoded quantities in visualizations. It plays a foundational role in understanding the relations between visual elements, such as the bar length ratios in bar charts. Zhenxing Cui is with the School of Computer Science and Technology, Shandong University, China. Lu Chen is with the State Key Lab of CAD&CG, Zhejiang University, China. Y unhai Wang is with the School of Information, Renmin University of China, China. Daniel Haehn is with the College of Science and Mathematics, University of Massachusetts Boston, USA.