This report presents the evaluation approach developed for the DARPA Big Mechanism program, which aimed at developing computer systems that will read research papers, integrate the information into a computer model of cancer mechanisms, and frame new hypotheses. We employed an iterative, incremental approach to the evaluation of the three phases of the program. In Phase I, we evaluated the ability of system and human teams ability to read-with-a-model to capture mechanistic information from the biomedical literature, integrated with information from expert curated biological databases. In Phase II we evaluated the ability of systems to assemble fragments of information into a mechanistic model. The Phase III evaluation focused on the ability of systems to provide explanations of experimental observations based on models assembled (largely automatically) by the Big Mechanism process. The evaluation for each phase built on earlier evaluations and guided developers towards creating capabilities for the new phase. The report describes our approach, including innovations such as a reference set (a curated data set limited to major findings of each paper) to assess the accuracy of systems in extracting mechanistic findings in the absence of a gold standard, and a method to evaluate model-based explanations of experimental data. Results of the evaluation and supporting materials are included in the appendices.

In open-retrieval conversational machine reading (OR-CMR) task, machines are required to do multi-turn question answering given dialogue history and a textual knowledge base. Existing works generally utilize two independent modules to approach this problem's two successive sub-tasks: first with a hard-label decision making and second with a question generation aided by various entailment reasoning methods. Such usual cascaded modeling is vulnerable to error propagation and prevents the two sub-tasks from being consistently optimized. In this work, we instead model OR-CMR as a unified text-to-text task in a fully end-to-end style. Experiments on the OR-ShARC dataset show the effectiveness of our proposed end-to-end framework on both sub-tasks by a large margin, achieving new state-of-the-art results. Further ablation studies support that our framework can generalize to different backbone models.

In conversational machine reading, systems need to interpret natural language rules, answer high-level questions such as "May I qualify for VA health care benefits?", and ask follow-up clarification questions whose answer is necessary to answer the original question. However, existing works assume the rule text is provided for each user question, which neglects the essential retrieval step in real scenarios. In this work, we propose and investigate an open-retrieval setting of conversational machine reading. In the open-retrieval setting, the relevant rule texts are unknown so that a system needs to retrieve question-relevant evidence from a collection of rule texts, and answer users' high-level questions according to multiple retrieved rule texts in a conversational manner. We propose MUDERN, a Multi-passage Discourse-aware Entailment Reasoning Network which extracts conditions in the rule texts through discourse segmentation, conducts multi-passage entailment reasoning to answer user questions directly, or asks clarification follow-up questions to inquiry more information. On our created OR-ShARC dataset, MUDERN achieves the state-of-the-art performance, outperforming existing single-passage conversational machine reading models as well as a new multi-passage conversational machine reading baseline by a large margin. In addition, we conduct in-depth analyses to provide new insights into this new setting and our model.

Conversational Machine Reading (CMR) aims at answering questions in a complicated manner. Machine needs to answer questions through interactions with users based on given rule document, user scenario and dialogue history, and ask questions to clarify if necessary. In this paper, we propose a dialogue graph modeling framework to improve the understanding and reasoning ability of machine on CMR task. There are three types of graph in total. Specifically, Discourse Graph is designed to learn explicitly and extract the discourse relation among rule texts as well as the extra knowledge of scenario; Decoupling Graph is used for understanding local and contextualized connection within rule texts. And finally a global graph for fusing the information together and reply to the user with our final decision being either ``Yes/No/Irrelevant" or to ask a follow-up question to clarify.

Microsoft researchers have already created technology that can do two difficult tasks about as well as a person: identify images and recognize words in a conversation. Now, the company's leading AI experts are working on systems that can do something even more complex: Read passages of text and answer questions about them. "We're trying to develop what we call a literate machine: A machine that can read text, understand text and then learn how to communicate, whether it's written or orally," said Kaheer Suleman, the co-founder of Maluuba, a Quebec-based deep learning startup that Microsoft acquired earlier this year. The Maluuba team is one of several groups at Microsoft that are tackling the challenge of machine reading. Two other research teams, one at the company's Redmond, Washington, headquarters and the other in its Beijing, China, research lab, are currently leading a competition run by Stanford University that uses information from Wikipedia to test how well AI systems can answer questions about text passages.

If you asked most people, they'd probably say that computers and other gadgets are pretty good at communicating information to us, whether it's by providing directions to an important business meeting or finding the best recipe for gluten-free apple pie. And yet, computers still don't communicate with us nearly as intuitively as we communicate with each other. If you type a query into a search engine, for example, chances are you'll get a list of websites to click on. But if you ask a person a question, she'll respond with an answer, or perhaps ask another question to get more information before answering. Microsoft is hoping to improve how well computers can communicate information to us.

The code has been copied to your clipboard. Machines equipped with artificial intelligence (AI) have performed better than human beings in a high-level test of reading comprehension. Two natural language processing tools received higher test scores than humans in recent exams. One of the tools is a product of the American software maker Microsoft. The other was created by the Chinese online seller Alibaba Group.

Microsoft researchers have already created technology that can do two difficult tasks about as well as a person: identify images and recognize words in a conversation. Now, the company's leading AI experts are working on systems that can do something even more complex: Read passages of text and answer questions about them. "We're trying to develop what we call a literate machine: A machine that can read text, understand text and then learn how to communicate, whether it's written or orally," said Kaheer Suleman, the co-founder of Maluuba, a Quebec-based deep learning startup that Microsoft acquired earlier this year. Machine reading systems also could help doctors, lawyers and other experts more quickly get through the drudgery of things like reading through documents for specific medical findings or rarified legal precedent. That would leave experts more time to focus on treating patients or formulating legal defenses.

Today is an exciting day for the advancement of AI at Microsoft. We have agreed to acquire Maluuba, a Montreal-based company with one of the world's most impressive deep learning research labs for natural language understanding. Maluuba's expertise in deep learning and reinforcement learning for question-answering and decision-making systems will help us advance our strategy to democratize AI and to make it accessible and valuable to everyone -- consumers, businesses and developers. We've recently set new milestones for speech and image recognition using deep learning techniques, and with this acquisition we are, as Wayne Gretzky would say, skating to where the puck will be next -- machine reading and writing. Maluuba's vision is to advance toward a more general artificial intelligence by creating literate machines that can think, reason and communicate like humans -- a vision exactly in line with ours.

Today is an exciting day for the advancement of AI at Microsoft. We have agreed to acquire Maluuba, a Montreal-based company with one of the world's most impressive deep learning research labs for natural language understanding. Maluuba's expertise in deep learning and reinforcement learning for question-answering and decision-making systems will help us advance our strategy to democratize AI and to make it accessible and valuable to everyone -- consumers, businesses and developers. We've recently set new milestones for speech and image recognition using deep learning techniques, and with this acquisition we are, as Wayne Gretzky would say, skating to where the puck will be next -- machine reading and writing. Maluuba's vision is to advance toward a more general artificial intelligence by creating literate machines that can think, reason and communicate like humans -- a vision exactly in line with ours.