Man and Machine

Engineers at Pinterest constantly create new artificial-intelligence algorithms to help its users find what they're looking for among billions of pictures of food, products, houses, and other items. Matching search queries with relevant images is crucial to keep users coming back. But until last year, it could take days to test the effectiveness of each new algorithm. To fine-tune its machine learning and provide better search results faster, Pinterest turned to an unexpected source: human intelligence. It hired crowdsourcing companies such as CrowdFlower to marshal people to quickly do "micro-tasks" such as labeling photos and assessing the quality of search results.