Scene understanding algorithms in computer vision are improving dramatically by training deep convolutional neural networks on millions of accurately annotated images. Collecting large-scale datasets for this kind of training is challenging, and the learning algorithms are only as good as the data they train on. Training annotations are often obtained by taking the majority label from independent crowdsourced workers using platforms such as Amazon Mechanical Turk. However, the accuracy of the resulting annotations can vary, with the hardest-to-annotate samples having prohibitively low accuracy. Our insight is that in cases where independent worker annotations are poor more accurate results can be obtained by having workers collaborate. This paper introduces consensus agreement games, a novel method for assigning annotations to images by the agreement of multiple consensuses of small cliques of workers. We demonstrate that this approach reduces error by 37.8% on two different datasets at a cost of $0.10 or $0.17 per annotation. The higher cost is justified because our method does not need to be run on the entire dataset. Ultimately, our method enables us to more accurately annotate images and build more challenging training datasets for learning algorithms.

Abstracts of the invited talks presented at the AAAI Fall Symposium on Discovery Informatics: AI Takes a Science-Centered View on Big Data. Talks include  A Data Lifecycle Approach to Discovery Informatics,  Generating Biomedical Hypotheses Using Semantic Web Technologies,  Socially Intelligent Science, Representing and Reasoning with Experimental and Quasi-Experimental Designs, Bioinformatics Computation of Metabolic Models from Sequenced Genomes, Climate Informatics: Recent Advances and Challenge Problems for Machine Learning in Climate Science,  Predictive Modeling of Patient State and Therapy Optimization, Case Studies in Data-Driven Systems: Building Carbon Maps to Finding Neutrinos,  Computational Analysis of Complex Human Disorders, and Look at This Gem: Automated Data Prioritization for Scientific Discovery of Exoplanets, Mineral Deposits, and More.

The Association for the Advancement of Artificial Intelligence was pleased to present the 2012 Fall Symposium Series, held Friday through Sunday, November 2–4, at the Westin Arlington Gateway in Arlington, Virginia. The titles of the eight symposia were as follows: AI for Gerontechnology (FS-12-01), Artificial Intelligence of Humor (FS-12-02), Discovery Informatics: The Role of AI Research in Innovating Scientific Processes (FS-12-03), Human Control of Bio-Inspired Swarms (FS-12-04), Information Retrieval and Knowledge Discovery in Biomedical Text (FS-12-05), Machine Aggregation of Human Judgment (FS-12-06), Robots Learning Interactively from Human Teachers (FS-12-07), and Social Networks and Social Contagion (FS-12-08). The highlights of each symposium are presented in this report.

The Association for the Advancement of Artificial Intelligence was pleased to present the 2012 Fall Symposium Series, held Friday through Sunday, November 2–4, at the Westin Arlington Gateway in Arlington, Virginia. The titles of the eight symposia were as follows: AI for Gerontechnology (FS-12-01), Artificial Intelligence of Humor (FS-12-02), Discovery Informatics: The Role of AI Research in Innovating Scientific Processes (FS-12-03), Human Control of Bio-Inspired Swarms (FS-12-04), Information Retrieval and Knowledge Discovery in Biomedical Text (FS-12-05), Machine Aggregation of Human Judgment (FS-12-06), Robots Learning Interactively from Human Teachers (FS-12-07), and Social Networks and Social Contagion (FS-12-08). The highlights of each symposium are presented in this report.

Addressing the ambitious research agendas put forward by many scientific disciplines requires meeting a multitude of challenges in intelligent systems, information sciences, and human-computer interaction. Many aspects of the scientific discovery process are often largely manual and could be automated, improved, or made more efficient. Better interfaces for collaboration, visualization, and understanding would significantly improve scientific practice. Scientific data, publications, and tools could be published in open formats with appropriate semantic descriptions and metadata annotations to improve sharing and dissemination. Opportunities for broader participation in well-defined scientific tasks enable human contributors to provide large amounts of data, annotations, or complex processing results that could not otherwise be obtained. Improvements and innovations across the spectrum of scientific processes and activities will have a profound impact on the rate of scientific discoveries.

In this paper, we present CrowdSense, an algorithm for estimating the crowd’s majority opinion by querying only a subset of it. CrowdSense works in an online fashion where examples come one at a time and it dynamically samples subsets of labelers based on an exploration/exploitation criterion. The algorithm produces a weighted combination of a subset of the labelers’ votes that approximates the crowd’s opinion. We also present two probabilistic variants of CrowdSense that are based on different assumptions on the joint probability distribution between the labelers’ votes and the majority vote. Our experiments demonstrate that we can reliably approximate the entire crowd’s vote by collecting opinions from a representative subset of the crowd.

Artificial Intelligence researchers have long sought to understand and replicate processes of scientific discovery. This article discusses Discovery Informatics as an emerging area of research that builds on that tradition and applies principles of intelligent computing and information systems to understand, automate, improve, and innovate processes of scientific discovery.

AAAI 2008 Workshop Reports

AAAI was pleased to present the AAAI-08 Workshop Program, held Sunday and Monday, July 13–14, in Chicago, Illinois, USA. The program included the following 15 workshops: Advancements in POMDP Solvers; AI Education Workshop Colloquium; Coordination, Organizations, Institutions, and Norms in Agent Systems, Enhanced Messaging; Human Implications of Human-Robot Interaction; Intelligent Techniques for Web Personalization and Recommender Systems; Metareasoning: Thinking about Thinking; Multidisciplinary Workshop on Advances in Preference Handling; Search in Artificial Intelligence and Robotics; Spatial and Temporal Reasoning; Trading Agent Design and Analysis; Transfer Learning for Complex Tasks; What Went Wrong and Why: Lessons from AI Research and Applications; and Wikipedia and Artificial Intelligence: An Evolving Synergy.

Artificial Intelligence: The Next Twenty-Five Years