In adversarial data collection (ADC), a human workforce interacts with a model in real time, attempting to produce examples that elicit incorrect predictions. Researchers hope that models trained on these more challenging datasets will rely less on superficial patterns, and thus be less brittle. However, despite ADC's intuitive appeal, it remains unclear when training on adversarial datasets produces more robust models. In this paper, we conduct a large-scale controlled study focused on question answering, assigning workers at random to compose questions either (i) adversarially (with a model in the loop); or (ii) in the standard fashion (without a model). Across a variety of models and datasets, we find that models trained on adversarial data usually perform better on other adversarial datasets but worse on a diverse collection of out-of-domain evaluation sets. Finally, we provide a qualitative analysis of adversarial (vs standard) data, identifying key differences and offering guidance for future research.

The oil and natural gas industry is turning to artificial intelligence technology to save billions of dollars in maintenance and production costs. Houston oilfield services company Baker Hughes, tech giant Microsoft and Silicon Valley artificial intelligence company C3.ai have signed an agreement to develop and deploy the technology for industry customers around the globe, the companies said Tuesday. In the oil field, artificial intelligence technology is being used to compile massive amounts of data transmitted by sensors and so-called smart equipment, look for patterns, make predictions and inform decisions by operators. "Companies that adopt this technology will be the next Amazon, and those that don't adopt will be the next Sears," Tom Siebel, C3.ai founder and CEO, said in an interview. Baker Hughes and C3.ai launched a joint venture in June to deploy artificial intelligence in the oil patch.