Beyond Mechanical Turk: Using Techniques from Meta Learning to Compare Crowdsourcing Platforms Across Languages

Successful natural language processing (NLP) annotation tasks utilize crowdsourced worker pools that are optimized for the quality, cost, and speed of task completion. Recent work has shown that it is difficult to compare worker pools across platforms because there are insufficient numbers of workers in the geographical locations speaking the languages that are required. This problem is only getting worse with limits on worker location on the dominant crowdsourced worker platform, Amazon’s Mechanical Turk. Preliminary work suggests a general method for measuring the cost in time, money, and worker quality across platforms. This paper suggests a strategy for comparing a series of annotation tasks that seek a more definite, empirical approach. This work in progress proposes running a series of parallel experiments across popular crowdsourcing platforms and using strategies from image processing tasks to compare task difficulty.