Do humans and machines have the same eyes? Human-machine perceptual differences on image classification

One motivation of neural networks (NN) is creating artificial intelligence that can learn from human intelligence and mimic human behavior. In computer vision, researchers often build their work upon the assumption that neural networks learn a feature representation similar to visual cortex activity [3, 53, 37]. It is believed that a well-trained network learns to represent input stimuli in a way that is similar to human visual perception [9]. As a result, most current work in computer vision that aims to develop better computer models focuses on benchmark scores (e.g., prediction accuracy) and ignores the evaluations of human-machine similarity. Figure 1: Upper: Our study aims to understand the perceptual difference between human and machine classifiers. Lower: We empirically demonstrate the benefit of utilizing their perceptual difference with a post-hoc collaboration.