Better Peer Grading through Bayesian Inference

Peer grading is a powerful pedagogical tool. It benefits students by giving them exposure to others' perspectives; helping them to internalize evaluation criteria by applying them critically to peer work Lu and Law (2012); and offering them feedback from equal-status learners Topping (2009). Just as importantly, it gives instructors a way to make classes more scalable by shifting (some) grading workload away from course staff; effectively, this again benefits students, by giving them more opportunities for their work to be evaluated. In order for peer grading systems to be both useful to instructors and acceptable to students, they must produce grades that are sufficiently similar to those that an instructor would have given. This is a challenging task because individual peer graders will be biased (consistently give generous or harsh grades); noisy (the same grader could grade an assignment differently on different days); and potentially strategic (some students will enter insincere peer grades unrelated to a submission's quality if they can get away with it). Addressing these interrelated challenges has been a topic of academic study in Computer Science for at least the last two decades. The first methods for aggregating peer grades--and many others introduced more recently--produce point estimates of each assignment's grade and each grader's quality (Walsh, 2014; Chakraborty et al., 2018; Prajapati et al., 2020; de Alfaro and Shavlovsky, 2014; Hamer et al., 2005). At their best, methods that produce point estimates maximize the likelihood of the data given a model, e.g., by assigning each grader a "reliability" parameter and iteratively updating these parameters to best describe the reported grades.