Enabling machines to understand structured visuals like slides and user interfaces is essential for making them accessible to people with disabilities. However, achieving such understanding computationally has required manual data collection and annotation, which is time-consuming and labor-intensive. To overcome this challenge, we present a method to generate synthetic, structured visuals with target labels using code generation. Our method allows people to create datasets with built-in labels and train models with a small number of human-annotated examples. We demonstrate performance improvements in three tasks for understanding slides and UIs: recognizing visual elements, describing visual content, and classifying visual content types.

The focus of HCOMP 2014 was the crowd worker. While crowdsourcing is motivated by the promise of leveraging people's intelligence and diverse skillsets in computational processes, the human aspects of this workforce are all too often overlooked. Instead, workers are frequently viewed as interchangeable components that can be statistically managed to eek out reasonable outputs.We are quickly moving past and rejecting these notions, and beginning to understand that it is sometimes the very abstractions that we introduce to make human computation feasible, e.g., abstracting humans behind APIs or isolating workers from others in order to ensure independent input, that can lead to the problems that we then set about trying to solve, e.g., poor or inconsistent quality work. Creating a brighter future for crowd work will require new socio-technical systems that not only decompose tasks, recruit and coordinate workers, and make sense of results, but also find interesting tasks for people to contribute to, structure tasks so that workers learn from them as they go, and eventually automate mundane parts of work. Research in artificial intelligence will be vital for achieving this future.