Davis, R.

Gesture Salience as a Hidden Variable for Coreference Resolution and Keyframe Extraction

Journal of Artificial Intelligence Research

Gesture is a non-verbal modality that can contribute crucial information to the understanding of natural language. But not all gestures are informative, and non-communicative hand motions may confuse natural language processing (NLP) and impede learning. People have little diffculty ignoring irrelevant hand movements and focusing on meaningful gestures, suggesting that an automatic system could also be trained to perform this task. However, the informativeness of a gesture is context-dependent and labeling enough data to cover all cases would be expensive. We present conditional modality fusion, a conditional hidden-variable model that learns to predict which gestures are salient for coreference resolution, the task of determining whether two noun phrases refer to the same semantic entity. Moreover, our approach uses only coreference annotations, and not annotations of gesture salience itself. We show that gesture features improve performance on coreference resolution, and that by attending only to gestures that are salient, our method achieves further significant gains. In addition, we show that the model of gesture salience learned in the context of coreference accords with human intuition, by demonstrating that gestures judged to be salient by our model can be used successfully to create multimedia keyframe summaries of video. These summaries are similar to those created by human raters, and significantly outperform summaries produced by baselines from the literature.

Diagnostic reasoning based on structure and behavior


See also:Retrospective on "Diagnostic reasoning based on structure and behavior", Artificial Intelligence (59) 1993 149-157MIT A.I. Memo 739, June, 1984Artificial Intelligence 24:347-410

Negotiation as a metaphor for distributed problem solving


"We describe the concept of distributed problem solving and define it as the cooperative solution of problems by a decentralized and loosely coupled collection of problem solvers. This approach to problem solving offers the promise of increased performance and provides a useful medium for exploring and developing new problem-solving techniques.We present a framework called the contract net that specifies communication and control in a distributed problem solver. Task distribution is viewed as an interactive process, a discussion carried on between a node with a task to be executed and a group of nodes that may be able to execute the task. We describe the kinds of information that must be passed between nodes during the discussion in order to obtain effective problem-solving behavior. This discussion is the origin of the negotiation metaphor: Task distribution is viewed as a form of contract negotiation.We emphasize that protocols for distributed problem solving should help determine the content of the information transmitted, rather than simply provide a means of sending bits from one node to another.The use of the contract net framework is demonstrated in the solution of a simulated problem in area surveillance, of the sort encountered in ship or air traffic control. We discuss the mode of operation of a distributed sensing system, a network of nodes extending throughout a relatively large geographic area, whose primary aim is the formation of a dynamic map of traffic in the area.From the results of this preliminary study we abstract features of the framework applicable to problem solving in general, examining in particular transfer of control. Comparisons with PLANNER, CONNIVER, HEARSAY-II, AND PUP6 are used to demonstrate that negotiation—the two-way transfer of information—is a natural extension to the transfer of control mechanisms used in earlier problem-solving systems."Artificial Intelligence: 20:63-109

Expert Systems: Where Are We? And Where Do We Go from Here?


"Work on Expert Systems has received extensive attention recently, prompting growing interest in a range of environments. Much has been made of the basic concept and of the rule-based system approach typically used to construct the programs. Perhaps this is a good time then to review what we know, asses the current prospects, and suggest directions appropriate for the next steps of basic research. I'd like to do that today, and propose to do it by taking you on a journey of sorts, a metaphorical trip through the State of the Art of Expert Systems. We'll wander about the landscape, ranging from the familiar territory of the Land of Accepted Wisdom, to the vast unknowns at the Frontiers of Knowledge. I guarantee we'll all return safely, so come along...."AI Magazine 3(2): Spring 1982, 3-22.

Frameworks for cooperation in distributed problem solving


"Two forms of cooperation in distributed problem solving are considered: task-sharing and result-sharing. In the former, nodes assist each other by sharing the computational load for the execution of subtasks of the overall problem. In the latter, nodes assist each other by sharing partial results which are based on somewhat different perspectives on the overall problem. Different perspectives arise because the nodes use different knowledge sources (KS’s) (e.g., syntax versus acoustics in the case of a speech-understanding system) or different data (e.g., data that is sensed at different locations in the case of a distributed sensing system). Particular attention is given to control and to internode communication for the two forms of cooperation. For each, the basic methodology is presented and systems in which it has been used are described. The two forms are then compared and the types of applications for which they are suitable are considered."Direct link to PDF.IEEE Transactions on Systems, Man, and Cybernetics, SMCll(l):61-70

Meta-rules: Reasoning about control


How can we insure that knowledge embedded in a program is applied effectively? Traditionally the answer to this question has been sought in different problem solving paradigms and in different approaches to encoding and indexing knowledge. Each of these is useful with a certain variety of problem, but they all share a common problem: they become ineffective in the face of a sufficiently large knowledge base. How then can we make it possible for a system to continue to function in the face of a very large number of plausibly useful chunks of knowledge?In response to this question we propose a framework for viewing issues of knowledge indexing and retrieval, a framework that includes what appears to be a useful perspective on the concept of a strategy. We view strategies as a means of controlling invocation in situations where traditional selection mechanisms become ineffective. We examine ways to effect such control, and describe meta-rules, a means of specifying strategies which offers a number of advantages. We consider at some length how and when it is useful to reason about control, and explore the advantages meta-rules offer for doing this. Artificial Intellligence 15:179-222

Interactive transfer of expertise: Acquisition of new inference rules


Summary of Ph.D. dissertation, Computer Science Dept., Stanford University (1979)."TEIRESIAS is a program designed to provide assistance on the task of building knowledge-based systems. It facilitates the interactive transfer of knowledge from a human expert to the system, in a high level dialog conducted in a restricted subset of natural language. This paper explores an example of TEIRESIAS in operation and demonstrates how it guides the acquisition of new inference rules. The concept of meta-level knowledge is described and illustrations given of its utility in knowledge acquisition and its contribution to the more general issues of creating an intelligent program."Also in:Readings in Artificial Intelligence, ed. Webber, Bonnie Lynn and Nils J. Nilsson, Palo Alto, CA: Tioga Publishing Co., 1981.Orig. in IJCAI-77, vol.1, pp. 321 ff. Preprint in Stanford HPP Report #HPP-77-9.See also: Artificial Intelligence, 12[#2]:409-427. Readings in Artificial Intelligence, ed. Webber, Bonnie Lynn and Nils J. Nilsson, Palo Alto, CA: Tioga Publishing Co., 1981

Meta-level knowledge: Overview and applications


"We define the concept of meta-level Knowledge, and illustrate it by briefly reviewing four examples that have been described in detail elsewhere. The examples include applications of the idea to tasks such as transfer of expertise from a domain expert to a program, and the maintenance and use of large Knowledge bases. We explore common themes that arise from these examples, and examine broader implications of the idea, in particular its impact on the design and construction of large programs."IJCAI 5, 920-927