Assessing the reproducibility of social science papers is essential for promoting rigor in research processes, but manual assessment is costly. With recent advances in agentic AI systems (i.e., AI agents), we seek to evaluate their capability to automate this process. However, existing benchmarks for reproducing research papers (1) focus solely on reproducing results using provided code and data without assessing their consistency with the paper, (2) oversimplify real-world scenarios, and (3) lack necessary diversity in data formats and programming languages. To address these issues, we introduce REPRO-Bench, a collection of 112 task instances, each representing a social science paper with a publicly available reproduction report. The agents are tasked with assessing the reproducibility of the paper based on the original paper PDF and the corresponding reproduction package. REPRO-Bench features end-to-end evaluation tasks on the reproducibility of social science papers with complexity comparable to real-world assessments. We evaluate three representative AI agents on REPRO-Bench, with the best-performing agent achieving an accuracy of only 21.4%. Building on our empirical analysis, we develop REPRO-Agent, which improves the highest accuracy achieved by existing agents by 71%. We conclude that more advanced AI agents should be developed to automate real-world reproducibility assessment. REPRO-Bench is publicly available at https://github.com/uiuc-kang-lab/REPRO-Bench.

The model proposed in the "Learn to Pay Attention" paper introduced a novel way to generate a trainable attention module for convolutional neural networks. The paper demonstrated the attention modulein VGG-based and ResNet-based architectures, provided several options for implementation (includingthree options for layer depths at which attention modules are to be implemented, two options for calculating the compatibility between the global and local features in generating the attention maps, and two options for what method will be used to produce output probabilities from global-level feature vectors), described the dataset preprocessing and model training routines, and reported the results of the consequent models in several tasks. We have successfully implemented allpossible configurations of both VGG-based and ResNet-based attention models, and have replicated the paper's reported results in image classification and fine-grain recognition task using the (VGG-att2)-concat-pc configuration on the CIFAR-10 dataset and (VGG-att3)-concatpc configurationon CIFAR-100 and the SVHN dataset.