Reviews: A Meta-Analysis of Overfitting in Machine Learning

In this paper the authors performed a large-scale empirical study of overfitting due to test set reuse in the machine learning community. The survey includes models from 112 Kaggle competitions and concluded that there is little evidence of substantial overfitting. The reviewers thought that the topic is important, that the paper was well written and easy to follow, and that the experiments are well executed. Since these concerns appeared late in the review process (reviewers were never given a chance to comment on the submissions' similarity), ultimately the PCs reviewed the situation. They determined that the submission was sufficiently different from 5286 and could be accepted.