Etzioni, Oren


Moving Beyond the Turing Test with the Allen AI Science Challenge

arXiv.org Artificial Intelligence

Given recent successes in AI (e.g., AlphaGo's victory against Lee Sedol in the game of GO), it's become increasingly important to assess: how close are AI systems to human-level intelligence? This paper describes the Allen AI Science Challenge---an approach towards that goal which led to a unique Kaggle Competition, its results, the lessons learned, and our next steps.


My Computer Is an Honor Student -- but How Intelligent Is It? Standardized Tests as a Measure of AI

AI Magazine

Given the well-known limitations of the Turing Test, there is a need for objective tests to both focus attention on, and measure progress towards, the goals of AI. In this paper we argue that machine performance on standardized tests should be a key component of any new measure of AI, because attaining a high level of performance requires solving significant AI problems involving language understanding and world modeling - critical skills for any machine that lays claim to intelligence. In addition, standardized tests have all the basic requirements of a practical test: they are accessible, easily comprehensible, clearly measurable, and offer a graduated progression from simple tasks to those requiring deep understanding of the world.


My Computer Is an Honor Student -- but How Intelligent Is It? Standardized Tests as a Measure of AI

AI Magazine

Given the well-known limitations of the Turing Test, there is a need for objective tests to both focus attention on, and measure progress towards, the goals of AI. In this paper we argue that machine performance on standardized tests should be a key component of any new measure of AI, because attaining a high level of performance requires solving significant AI problems involving language understanding and world modeling - critical skills for any machine that lays claim to intelligence. In addition, standardized tests have all the basic requirements of a practical test: they are accessible, easily comprehensible, clearly measurable, and offer a graduated progression from simple tasks to those requiring deep understanding of the world. Here we propose this task as a challenge problem for the community, summarize our state-of-the-art results on math and science tests, and provide supporting datasets


Identifying Meaningful Citations

AAAI Conferences

We introduce the novel task of identifying important citations in scholarly literature, i.e., citations that indicate that the cited work is used or extended in the new effort. We believe this task is a crucial component in algorithms that detect and follow research topics and in methods that measure the quality of publications.We model this task as a supervised classification problem at two levels of detail: a coarse one with classes (important vs. non-important), and a more detailed one with four importance classes. We annotate a dataset of approximately 450 citations with this information, and release it publicly. We propose a supervised classification approach that addresses this task with a battery of features that range from citation counts to where the citation appears in the body of the paper, and show that,our approach achieves a precision of 65% for a recall of 90%.


Commonsense from the Web: Relation Properties

AAAI Conferences

When general purpose software agents fail, it's often because they're brittle and need more background commonsense knowledge. In this paper we present relation properties as a valuable type of commonsense knowledge that can be automatically inferred at scale by reading the Web. People base many commonsense inferences on their knowledge of relation properties such as functionality, transitivity, and others. For example, all people know that bornIn(Year) satisfies the functionality property, meaning that each person can be born in exactly one year. Thus inferences like "Obama was born in 1961, so he was not born in 2008", which computers do not know, are obvious even to children. We demonstrate scalable heuristics for learning relation functionality from noisy Web text that outperform existing approaches to detecting functionality. The heuristics we use address Web NLP challenges that are also common to learning other relation properties, and can be easily transferred. Each relation property we learn for a Web-scale set of relations will enable computers to solve real tasks, and the data from learning many such properties will be a useful addition to general commonsense knowledge bases.


Adapting Open Information Extraction to Domain-Specific Relations

AI Magazine

Information extraction (IE) can identify a set of relations from free text to support question answering (QA). Until recently, IE systems were domain-specific and needed a combination of manual engineering and supervised learning to adapt to each target domain. A new paradigm, Open IE operates on large text corpora without any manual tagging of relations, and indeed without any pre-specified relations. We explore the steps needed to adapt Open IE to a domain-specific ontology and demonstrate our approach of mapping domain-independent tuples to an ontology using domains from DARPA's Machine Reading Project.


Adapting Open Information Extraction to Domain-Specific Relations

AI Magazine

Information extraction (IE) can identify a set of relations from free text to support question answering (QA). Until recently, IE systems were domain-specific and needed a combination of manual engineering and supervised learning to adapt to each target domain. A new paradigm, Open IE operates on large text corpora without any manual tagging of relations, and indeed without any pre-specified relations. Due to its open-domain and open-relation nature, Open IE is purely textual and is unable to relate the surface forms to an ontology, if known in advance. We explore the steps needed to adapt Open IE to a domain-specific ontology and demonstrate our approach of mapping domain-independent tuples to an ontology using domains from DARPA’s Machine Reading Project. Our system achieves precision over 0.90 from as few as 8 training examples for an NFL-scoring domain.


Panlingual Lexical Translation via Probabilistic Inference

AAAI Conferences

The bare minimum lexical resource required to translate between a pair of languages is a translation dictionary. Unfortunately, dictionaries exist only between a tiny fraction of the 49 million possible language-pairs making machine translation virtually impossible between most of the languages. This paper summarizes the last four years of our research motivated by the vision of panlingual communication. Our research comprises three key steps. First, we compile over 630 freely available dictionaries over the Web and convert this data into a single representation – the translation graph. Second, we build several inference algorithms that infer translations between word pairs even when no dictionary lists them as translations. Finally, we run our inference procedure offline to construct PANDICTIONARY– a sense-distinguished, massively multilingual dictionary that has translations in more than 1000 languages. Our experiments assess the quality of this dictionary and find that we have 4 times as many translations at a high precision of 0.9 compared to the English Wiktionary, which is the lexical resource closest to PANDICTIONARY.


AAAI 2007 Spring Symposium Series Reports

AI Magazine

The 2007 Spring Symposium Series was held Monday through Wednesday, March 26-28, 2007, at Stanford University, California. The titles of the nine symposia in this symposium series were (1) Control Mechanisms for Spatial Knowledge Processing in Cognitive/Intelligent Systems, (2) Game Theoretic and Decision Theoretic Agents, (3) Intentions in Intelligent Systems, (4) Interaction Challenges for Artificial Assistants, (5) Logical Formalizations of Commonsense Reasoning, (6) Machine Reading, (7) Multidisciplinary Collaboration for Socially Assistive Robotics, (8) Quantum Interaction, and (9) Robots and Robot Venues: Resources for AI Education.


AAAI 2007 Spring Symposium Series Reports

AI Magazine

The 2007 Spring Symposium Series was held Monday through Wednesday, March 26-28, 2007, at Stanford University, California. The titles of the nine symposia in this symposium series were (1) Control Mechanisms for Spatial Knowledge Processing in Cognitive/Intelligent Systems, (2) Game Theoretic and Decision Theoretic Agents, (3) Intentions in Intelligent Systems, (4) Interaction Challenges for Artificial Assistants, (5) Logical Formalizations of Commonsense Reasoning, (6) Machine Reading, (7) Multidisciplinary Collaboration for Socially Assistive Robotics, (8) Quantum Interaction, and (9) Robots and Robot Venues: Resources for AI Education.