Mitigating Cognitive Biases in Multi-Criteria Crowd Assessment

Despite recent advances in AI and machine learning technologies, many applications still require human assessment because the characteristics of objects that can explain human subjectivity are sometimes unknown or too vague to be extracted automatically, which is a serious bottleneck when conducting large-scale automated quality assessments. The use of crowdsourcing is a promising way to implement this with the wisdom of the crowd. One challenge in crowdsourced quality assessments is the uncertainty of human judgments. Since workers have different competences, expertise, or motivations, their responses are sometimes too noisy to analyze and extract useful knowledge. A straightforward solution is to assign multiple crowdworkers to each evaluation target and aggregate the redundantly collected evaluations using majority voting. More sophisticated statistical methods, such as Bayesian generative models, have also been explored for better aggregations. Various factors of human error have been introduced into statistical models, such as the ability of workers (Dawid & Skene, 1979), difficulty of the questions (Whitehill et al., 2009; Welinder et al., 2011), and presence of malicious workers (Raykar & Yu, 2011).