This paper is the author's contribution to Chapter 6. in the volume BU:LDIN3 EXPERT SYSTEMS, edited by R. Hayes-Roth, D. Lenat, and D. Waterman:4 The full article is entitled "Evaluation of expert systems: issues and case studies", and is authored by J. Gaschnig, P. Klahr, H. Pople, E. Shortliffe. The volume is the result of a Workshop on Expert Systems held in San Diego in August 1980 and sponsored by the Rand Corporation, ARPA, and the NSF. Parts of Chapters 7 & 8. Reprinted with permission. In this section we define ma,ly of the parameters that determine an appropriate design for an evaluation experiment. When one examines the literature on computer performance evaluation, it is clear that the term is used with a variety of meanings depending upon the;.-c.3pective of tne authors. Each tends to focus on the specific performance issues that have been most central to the design of the system in question. Utner aspects warranting formal evaluation are often ignored.

Reprinted from IJCAI, August 24-28, 1981, Vancouver, British Columbia. Used by permission of the International Joint Conference on Artificial Intelligence, Inc.; copies of the Proceedings are available from Morgan Kaufmann Publishers, Inc., 95 First Street, Los Altos, CA 94022, USA. DART: An Expert System for Computer Fault Diagnosis James S. Bennett Heuristic Programming Project, Computer Science Deparunent Stanford University, Stanford, CA 94305 Clifford R. Hollander IBM Scientific Center, 1530 Page Mill Road Palo Alto, CA 94304 To appear in International Joint Conference on Artificial Intelligence, August 24-28, 1981, Vancouver, British Columbia. A. Intro Juction We describe an application of artificial intelligence techniques to computer system fault diagnosis, in particular, we have implemented an automated consultant that advises IBM field service personnel on the diagnosis of faults occurring in computer installations. The consultant identifies specific system components (both hardware and software) likely to be responsible for an observed fault and offers a brief explanation of the major factors and evidence supporting these indictments.

We describe an oncology protocol management system, named ONCOCIN, that is designed to assist physicians in the treatment of cancer patients. The system is a set of related programs, one of which is a rule-based reasoner that encompasses the necessary knowledge of cancer chemotherapy. Representation and control techniques are dizcussed, and ONCOCIN is contra7ted with:qstems that could be built using EMYCIN. Of particular Interest is the need to provide ONCOCIN with an interface that will make the system acceptable to oncologists.

We describe a program for verifying that a set of rules in an expert system comprehensively spans the knowledge of a specialized domain. The program has been devised and tested within the context of the ONCOCIN System, s rule-- based consultant for clinical oncology. The stylized format of ONCOCTN's rules has allowed the automatic detection of a number of common errors as the knowledge base has been developed. This capability suggests a general mecharism for correcting most problems with knowledge base completeness and consistency before they can cause performance errors.

The following maxims represent a distillat:on of some of these intuitions and heuristics. They are not necessarily full of great insight. In many ways, they are similar to well-known guidelines for building other types of software. But we give them here with the hope that they will be helpful to future knowledge engineers.

The Key idea is to represent explicitly and separately: a A. The limitations of MYCIN for application to teaching domain-independent diagnostic strategy in the form of meta-rules,

Expert Systems constitute a subclass of Al reasoning programs which are distinguished by criteria of usefulness and understandability as well as performance. In this paper these criteria are discussed, the state of the art of so-called "Level-1" Expert Systems is assessed, and the research topics necessary for moving to Level-2 systems are reviewed.

To be presented at the Eighteenth Annual Meeting of the Drug Information Association, Kansas City, Missouri, June 16, 1982. Edward H. Shortliffe, MD, PhD Assistant Professor of Medicine and Computer Science Heuristic Programming Project Department of Medicine Stanford University School of Medicine Stanford, California 94305 To be presented at the Eighteenth Annual Meeting of the Drug Information Association Kansas City, Missouri 16 June 1982 ABSTRACT The trend towards increased use of computer-based symbolic reasoning techniques for clinical decision making programs stems from the dual goals of improving the performance and increasing the acceptance of such systems. This talk will summarize the design considerations that have encouraged some recent investigators to turn to artificial intelligence techniques when'bJllding consultation systems. Some of the recent experimental consultation systems are less concerned with reaching correct diagnoses than with advising physicians un optimal treatment strategies for patients with known serious ..',Iscases. Examples for discussion will be drawn from (1) the MYCIN system, a consultation program to advise physicians on the selection of antimicrobials for patients with bacteremia or meningitis, and (2) ONCOCIN, a recently developed program for advising oncologists on therapy adjustment in the manarement of patients enrolled in cancer chemotherapy protocols.

It is frequently observed that medicine.

This wolk was supported in parts by the Defense Advanced Rz-search Projects under ARPA contract no. MDA 903-77-C-0322 and National Institutes of He...Ith grant no.