Debiasing Evaluations That are Biased by Evaluations

It is common to aggregate information and evaluate items by collecting ratings on these items from people. In this work, we focus on the bias introduced by people's observable outcome or experience from the entity under evaluation, and we call it the "outcome-induced bias". Let describe this notion of bias with the help of two common applications - teaching evaluation and peer review. Many universities use student ratings for teaching evaluation. However, numerous studies have shown that student ratings are affected by the grading policy of the instructor [16, 26, 5]. For instance, as noted in [26, Chapter 4]: "...the effects of grades on teacher-course evaluations are both substantively and statistically important, and suggest that instructors can often double their odds of receiving high evaluations from students simply by awarding A's rather than B's or C's." As a consequence, the association between student ratings and teaching effectiveness can become negative [5], and student ratings serve as a poor predictor on the follow-on course achievement of the students [8, 6]: "...teachers who are associated with better subsequent performance receive worst evaluations from their students."