Octopus: A Framework for Cost-Quality-Time Optimization in Crowdsourcing

Task control of workflows over micro-task crowdsourcing platforms, such as Amazon Mechanical Turk (AMT), has received significant attention in AI literature (Weld et al. 2015). Typically, a requester needs to balance three competing objectives - (1) total cost, owing to payments made to workers for their responses (or ballots), (2) overall quality, usually evaluated as accuracy of the final output, and (3) the total time for completing the task. These criteria are interrelated: increasing the pay per task attracts more workers to the task, thereby reducing completion time. However, it also exhausts the budget sooner, so requesters can afford fewer ballots per task, likely reducing the overall quality. Most prior work on crowd controllers has focused on the tradeoff between cost (or no. of ballots) and quality (Dai et al. 2013; Lin, Mausam, and Weld 2012; Bragg, Mausam, and Weld 2013; Kamar et al. 2013; Parameswaran et al. 2012). A common approach is to define a Partially Observable Markov Decision Process (POMDP) per task, which decides on whether to get another ballot or submit the best answer for that task. However, this work is time-agnostic, and assumes that pay per ballot is given as input.