Appendix

The following section is answers to questions listed in datasheets for datasets. A.1 Motivation For what purpose was the dataset created? VisAlign is created to serve as a benchmark for measuring visual perception alignment between AI models and humans. Who created the dataset (e.g., which team, research group) and on behalf of which entity (e.g., company, institution, organization)? Who funded the creation of the dataset? If there is an associated grant, please provide the name of the grantor and the grant name and number. This work was supported by Institute of Information & Communications Technology Planning & Evaluation (IITP) grant (No.2019-0-00075, Artificial Intelligence Graduate School Program(KAIST)) and National Research Foundation of Korea (NRF) grant (NRF2020H1D3A2A03100945), funded by the Korea government (MSIT). A.2 Composition What do the instances that comprise the dataset represent (e.g., documents, photos, people, countries)? VisAlign contains eight different types of images and their corresponding gold human labels. How many instances are there in total (of each type, if appropriate)? There are a total of 12500 images in the train set, distributed equally among the 10 classes. The open test set and the closed test each contain 900 images: 100 images each in Categories 1 to 7 and 200 images in Category 8. Does the dataset contain all possible instances or is it a sample (not necessarily random) of instances from a larger set?