A major result of this research has been the development of a computer program that is intended to serve as a consultant in a number of medical problem areas. Here the considerations that underlie the program are discussed. The basic functions of the program are outlined in a nontechnical way, and an example of the use of the program is given. Then the results of the use of the program for several different medical problems are reviewed. Finally, an attempt is made to ascertain the potential of programs such as this in the delivery of appropriate medical care. Detailed reports on various aspects of this research are available in the literature (Gorry, 1967; 1968; Gorry and Barnett, 1968a; 1968b), and so the emphasis here will be on providing a general overview of the work and results obtained to date.

Every year, as computers become more powerful and less expensive, increasing amounts of health care data are recorded on them. Motivation for collecting data routinely into ambulatory and hospital medical record systems comes from all quarters. Health practitioners require sets of data for clinical management of individual patients. Hospital administrators require them for billing and resource allocation.

Recent research in clinical diagnosis (Barrows et al., 1978; Elstein et al., 1978; McGuire and Bashook, 1978) contributed to a consensus about the general form of the process of clinical diagnostic reasoning. Cues in patient data suggest hypotheses, which are, in turn, tested against subsequent data of the case. The basic hypothetico-deductive process is shared by experienced and inexperienced diagnosticians alike, as are numerous parametric characteristics of the process, such as the percentage of data items to first hypotheses, the average number of hypotheses maintained in active consideration, etc. These studies, however, have generally neglected the content of diagnostic reasoning, that is, the knowledge base of medical subject matter involved in the diagnostic process. Yet, despite prevalent findings of lack of differences in the form of diagnostic reasoning as a function of experience, the few differential findings from these research efforts implicate the importance of the knowledge base.

This chapter briefly outlines the difference between traditional instructional programs and ICAI. It then illustrates how GUIDON makes contributions in areas important to medical CAl: interacting with the student in a mixed-initiative dialogue (including the problems of feedback and realism), teaching problem-solving strategies, and assembling a computerbased curriculum. In evaluating GUIDON's performance, one can see the value in the basic idea of formalizing teaching knowledge in procedures that are separate from the knowledge to be taught. However, the program is inherently limited by the MYCIN knowledge base. The rule set is poorly structured, does not contain pathophysiological knowledge for justifying the diagnostic associations, and does not explicitly state the strategies for gathering information and focusing on hypotheses.

Since the early 1970s, researchers in computer-based medical reasoning have begun to recognize the potential benefits of applying symbolic reasoning techniques in clinical domains (see Chapter 3). One such research group is the Heuristic Programming Project at Stanford University. The first medical reasoning program developed by the project, known as the MYCIN system (Shortliffe, 1976), adopted symbolic processing techniques largely in response to a conviction that computer-based consultation systems, in order to be accepted by physicians, should be able to explain how and why a particular conclusion has been derived. Such systems should also be able to incorporate, organize, manipulate, and update large quantities of medical knowledge. Subsequently, a series of additional medical application programs using MYCIN's techniques has been created.

Turing's test (see evaluation) smart instruments, 345

In this book we have presented experimental evidence at many levels of detail for a diverse set of hypotheses. As indicated by the chapter and section headings, the major themes of the MYCIN work have many variations. In this final chapter we will try to summarize the most important results of the work presented. This recapitulation of the lessons learned should not be taken as a substitute for details in the sections themselves. We provide here an abstraction of the details, but hope it also constitutes a useful set of lessons on which others can build. The three main sections of this chapter will reiterate the main goals that provide the context for the experimental work; discuss the experimental results from each of the major parts of the book; and summarize the key questions we have been asked, or have asked ourselves, about the lessons we have learned. If we were to try to summarize in one word why MYCIN works as well as it does, that word would be flexibility.

Effect of the Tutorial The tutorial experience had a small but significant effect on physicians' Demands and also produced a substantial increase in their knowledge about computing concepts. The results from the Demand-scale were of particular interest. Physicians apparently gained new insights from the tutorial into the potential use and capabilities of medical computing and increased their performance Demands accordingly. These opinions regarding the attributes of acceptable computing systems were surprisingly uniform across physician subgroups both before and after the tutorial. Our interpretation of this result is that physicians are serious about these Demands and that consultation systems are not likely to be clinically effective, regardless of the accuracy of their advice, until these capabilities have been incorporated.

In August of 1980 Stanford hosted the annual Workshop on Artificial Intelligence in Medicine, and we organized a twoday tutorial program so that local physicians who were interested could learn about this emerging discipline. In addition, funding from the Henry J. Kaiser Family Foundation allowed us to support a questionnaire-based project to assess physicians' attitudes. Finally, a doctoral student in educational psychology, Randy Teach, joined the project that summer and brought with him much-needed skills in the areas of statistics, study design, and the use of computer-based statistical packages. The resulting study used the physicians who were attending the AIM tutorial as subjects, with a control group of M.D.'s drawn from the surrounding community. Chapter 34 summarizes the results and concludes with design recommendations derived from the data analysis. The reader is referred to that chapter for details; however, it is pertinent to reiterate here that a program's ability to give explanations for its reasoning was judged to be the single most important requirement for an advice-giving system in medicine. This observation accounts for our continued commitment to research on explanation, both in the ONCOCIN program (Langlotz and Shortliffe, 1983) and in current doctoral dissertations from the Heuristic Programming Project (Cooper, 1984; Kunz, 1984).

The three MYCIN studies, plus the designs for ONCOCIN evaluations that are nearing completion, have taught us many lessons about the validation of these kinds of programs. We summarize some of those points here in an effort to provide guidelines of use to others doing this kind of work.