The Winograd Schema Challenge

AAAI Conferences

In this paper, we present an alternative to the Turing Test that has some conceptual and practical advantages. A Winograd schema is a pair of sentences that differ only in one or two words and that contain a referential ambiguity that is resolved in opposite directions in the two sentences. We have compiled a collection of Winograd schemas, designed so that the correct answer is obvious to the human reader, but cannot easily be found using selectional restrictions or statistical techniques over text corpora. A contestant in the Winograd Schema Challenge is presented with a collection of one sentence from each pair, and required to achieve human-level accuracy in choosing the correct disambiguation.


Planning, Executing, and Evaluating the Winograd Schema Challenge

AI Magazine

The Winograd Schema Challenge (WSC) was proposed by Hector Levesque in 2011 as an alternative to the Turing test. Chief among its features is a simple question format that can span many commonsense knowledge domains. Questions are chosen so that they do not require specialized knoweldge or training and are easy for humans to answer. This article details our plans to run the WSC and evaluate results. Turing (1950) had first introduced the notion of testing a computer system's intelligence by assessing whether it could fool a human judge into thinking that it was conversing with a human rather a computer.


Winograd Schema Challenge Results: AI Common Sense Still a Problem, for Now

IEEE Spectrum Robotics

After a chatbot pretending to be a 13-year-old named Eugene Goostman "passed" a Turing test a few years ago, experts in artificial intelligence got together and decided that a traditional Turing test might not be all that effective in measuring the intelligence of a computer program after all. Instead, they came up with (among many other things) the Winograd Schema Challenge, which is intended to determine how well an artificial intelligence system handles commonsense reasoning: understanding the basics about how the world works, and implementing that knowledge in useful and accurate ways. A few weeks ago, the very first Winograd Schema Challenge took place at the International Joint Conference on Artificial Intelligence in New York City. We spoke with Charlie Ortiz, director of the Laboratory for AI and Natural Language Processing at Nuance Communications and one of the organizers of the Winograd Schema Challenge, about how things went, why the challenge is important, and what it means for the future of AI. The Winograd Schema Challenge tasks computer programs with answering a specific type of simple, commonsense question called a pronoun disambiguation problem (PDP).


An Approach to Solve Winograd Schema Challenge Using Automatically Extracted Commonsense Knowledge

AAAI Conferences

The Winograd Schema Challenge has recently been proposed as an alternative to the Turing test. A Winograd Schema consists of a sentence and question pair such that the answer to the question depends on the resolution of a definite pronoun in the sentence. The answer is fairly intuitive for humans but is difficult for machines because it requires commonsense knowledge about words or concepts in the sentence. In this paper we propose a novel technique which semantically parses the text, hunts for the needed commonsense knowledge and uses that knowledge to answer the given question.


Planning, Executing, and Evaluating the Winograd Schema Challenge

AI Magazine

The Winograd Schema Challenge was proposed by Hector Levesque in 2011 as an alternative to the Turing Test. Chief among its features is a simple question format that can span many commonsense knowledge domains. Questions are chosen so that they do not require specialized knoweldge or training, and are easy for humans to answer. This article details our plans to run the WSC and evaluate results.