Infusing Human Factors into Algorithmic Crowdsourcing

The emergence of crowdsourcing systems have provided a viable mechanism for incorporating humans into the computational loop at large scale and in real-time. This offers an unprecedent opportunity to study how artificial intelligence (AI) techniques and humans can collaborate to solve problems. An important challenge in crowdsourcing is how to make optimal use of human resources as people have different skills and their availability may be limited. In this paper, we provide the research community with a new dataset derived from an online game-based platform to address this challenge. Six crowdsourcing task allocation scenarios with different overall workload levels and worker population characteristics were presented to over 400 players to solve. With close to 3,000 game sessions and over 300,000 task allocation decisions from human and AI players, the dataset provides an efficient focal point for the research community to design solutions that can sustainably tap into the pool of human resources through crowdsourcing.