MobileWorks: Designing for Quality in a Managed Crowdsourcing Architecture (Extended Abstract)

Online labor marketplaces offer the potential to automate a variety of tasks too difficult for computers, but present requesters with significant difficulties in obtaining accurate results. We share experiences from building MobileWorks, a crowd platform that departs from the marketplace model to provide robust, high-quality results. Three architectural contributions yield measurably improved accuracy on input tasks.  A dynamic work routing system identifies expertise in the crowd and ensures that all work posted into the system is completed with bounded completion times and at fair worker prices. A peer management system ensures that incorrect answers are prevented by experienced members of the crowd. Last, social interaction techniques give the best workers the ability and incentives to manage, teach & supervise other members of the crowd, as well as to clarify tasks. This process filters worker error and allows the crowd to collectively learn how to solve unfamiliar tasks. (extended abstract)