Computational Support for Academic Peer Review

Peer review is the process by which experts in some discipline comment on the quality of the works of others in that discipline. Peer review of written works is firmly embedded in current academic research practice where it is positioned as the gateway process and quality control mechanism for submissions to conferences, journals, and funding bodies across a wide range of disciplines. It is probably safe to assume that peer review in some form will remain a cornerstone of academic practice for years to come, evidence-based criticisms of this process in computer science22,32,45 and other disciplines23,28 notwithstanding. While parts of the academic peer review process have been streamlined in the last few decades to take technological advances into account, there are many more opportunities for computational support that are not currently being exploited. The aim of this article is to identify such opportunities and describe a few early solutions for automating key stages in the established academic peer review process. When developing these solutions we have found it useful to build on our background in machine learning and artificial intelligence: in particular, we utilize a feature-based perspective in which the handcrafted features on which conventional peer review usually depends (for example, keywords) can be improved by feature weighting, selection, and construction (see Flach17 for a broader perspective on the role and importance of features in machine learning). Twenty-five years ago, at the start of our academic careers, submitting a paper to a conference was a fairly involved and time-consuming process that roughly went as follows: Once an author had produced the manuscript (in the original sense, that is, manually produced on a typewriter, possibly by someone from the university's pool of typists), he or she would make up to seven photocopies, stick all of them in a large envelope, and send them to the program chair of the conference, taking into account that international mail would take 3–5 days to arrive. On their end, the program chair would receive all those envelopes, allocate the papers to the various members of the program committee, and send them out for review by mail in another batch of big envelopes. Reviews would be completed by hand on paper and mailed back or brought to the program committee meeting. Finally, notifications and reviews would be sent back by the program chair to the authors by mail. Submissions to journals would follow a very similar process.