Experiments with the ICML 2020 peer-review process

The International Conference on Machine Learning (ICML) is a flagship machine learning conference that in 2020 received 4,990 submissions and managed a pool of 3,931 reviewers and area chairs. Given that the stakes in the review process are high -- the careers of researchers are often significantly affected by the publications in top venues -- we decided to scrutinize several components of the peer-review process in a series of experiments. Specifically, in conjunction with the ICML 2020 conference, we performed three experiments that target: resubmission policies, management of reviewer discussions, and reviewer recruiting. In this post, we summarize the results of these studies. Several leading ML and AI conferences have recently started requiring authors to declare previous submission history of their papers.