Quality Control in Crowdsourcing: An Objective Measurement Approach to Identifying and Correcting Rater Effects in the Social Evaluation of Products and Services

Crowdsourcing requires new strategies to evaluate the workers involved as well as the quality of workers’ output. Using customer feedback data, we introduce multi-facetted Rasch scaling as an evaluation technique to assess the contributions of workers and products simultaneously within a single coherent measurement framework. Based on a data set of about 250,000 customers who rated nearly 115,000 products, for a total of nearly 3 million cases, we found that product ratings reflect almost as much the existence of stable rater differences as they are indicative of the products’ properties. We illustrate how Rasch scaling provides extensive quality control mechanisms; as well we show how aberrant workers and products can be identified so that appropriate feedback and/or corrective actions can be initiated.