The three MYCIN studies, plus the designs for ONCOCIN evaluations that are nearing completion, have taught us many lessons about the validation of these kinds of programs. We summarize some of those points here in an effort to provide guidelines of use to others doing this kind of work.

A number of computer programs have been developed to assist physicians with diagnostic or treatment decisions, and many of them are potentially very useful tools. However, few systems have undergone evaluation by independent experts. The task evaluated was the selection of antimicrobials for cases of acute infectious meningitis before the causative agent was identified. MYCIN was originally developed in the domain of bacteremias and then expanded to include meningitis. Its task is a complicated one; it must decide whether and how to treat a patient, often in the absence of microbiological evidence. It must allow for the possibility that any important piece of information might be unknown or uncertain.

Here we consider the logical bases for rules: what kinds of arguments justify the rules, and what is their relation to a mechanistic model of the domain? We use the terms "explain" and "justify" synonymously, although the sense of "making clear what is not understood" (explain) is intended more than "vindicating, showing to be right or lawful" (justify).

This chapter explores a number of issues involving representation and use of what we term meta-level knowledge, or knowledge about knowledge.1 It begins by defining the term, then exploring a few of its varieties and considering the range of capabilities it makes possible. Four specific examples of meta-level knowledge are described, and a demonstration given of their application to a number of problems, including interactive transfer of expertise and the "intelligent" use of knowledge. Finally, we consider the long-term implications of the concept and its likely impact on the design of large programs. The context of this work is the TEIRESIAS program discussed in Chapter 9. In the earlier chapter we focused on the use of TEIRESIAS for knowledge acquisition. Here we focus on the classification and types of knowledge used by TEIRESIAS. In the most general terms, meta-level knowledge is knowledge about knowledge. Its primary use here is to enable a program to "know what it knows," and to make multiple uses of its knowledge. As mentioned in Chapter 9, the program is not only able to use its knowledge directly, but may also be able to examine it, abstract it, reason about it, or direct its application. This chapter discusses examples of meta-level knowledge classified along two dimensions: (i) specificity character (representation-specific vs. domain-specific), and (ii) source (user-supplied vs. derived). Representation-specific meta-level knowledge involves supplying a program with a store of knowledge dealing with the form of its representations, in particular, their design and organization. Traditionally, this design and organization infor-This chapter is an expanded and edited version of a paper originally appearing in Proceedings of the Fifth IJCAL 1977, pp. Used by permission of International Joint Conferences on Artificial Intelligence, Inc.; copies of the Proceedings are available from William Kaufmann, Inc., 95 First Street, Los Altos, CA 94022. IFollowing standard usage, knowledge about objects and relations in a particular domain will be referred to as object-level knowledge. Type declarations are a small step toward more explicit specification of this information, especially as they are used in extended data types and record structures. As we discuss below, this sort of information, along with a range of other facts about representation design, can be employed quite usefully if it is made explicit and made available to the system.

Performance Level D-RULE578 IF: 1) The infection which requires therapy is meningitis, and 2) Organisms were not seen on the stain of the culture, and 3) The type of the infection is bacterial, and 4) The patient has been seriously burned THEN: There is suggestiv evidence (.5) that pseudomonas-aeruginosa is one of the organisms (other than those seen on cultures or smears) which might be causing the infection UPDATES: COVERFOR USES: (TREATINF ORGSEEN TYPE BURNED) Support Level MECHANISM-FRAME: BODY-INFRACTION.WOUNDS JUSTIFICATION: "For a very brief period of time after a severe burn the surface of the wound is sterile. Shortly thereafter, the area becomes colonized by a mixed flora in which gram-positive organisms predominate. By the 3rd post-burn day this bacterial population becomes dominated by gram-negative organisms. By the 5th day these organisms have invaded tissue well beneath the surface of the burn. The organisms most commonly isolated from burn patients are Pseudomonas, Klebsiella-Enterobacter, Staph., etc. Infection with Pseudomonas is frequently fatal." LITERATURE: MacMillan BG: Ecology of Bacteria Colonizing the Burned Patient Given Topical and System Gentamicin Therapy: a five-year study, J Infect Dis 124:278-286, 1971.

We have so far described MYCIN largely in terms of its knowledge base and inference mechanism, and specifically in terms of rules and a rule interpreter that allow high-performance problem solving. In Chapters 27 through 29 we describe additional knowledge structures that increase the flexibility and transparency of' MYCIN's knowledge base. We refer to many of these as meta-level knowledge. When we speak of meta-level knowledge we mean nothing more than knowledge about knowledge. In a computer program it needs to be represented and interpreted in order to be useful, but the main idea is that it can be an explicit, and flexible, element of expertise. For example, metalevel knowledge can help in modifying an existing rule and in integrating the modification into the whole rule set because it provides additional information about the existing rules to the editor. The ideas for using meta-level knowledge in MYCIN grew out of several projects that Randy Davis was working on in the mid-1970s.

The idea of directly teaching students "how to think" goes back at least to Polya (1957), if not to Socrates, but it reached a new stage of development in Papert's laboratory (Papert, 1970). In the LOGO lab, young students were taught AI concepts such as hierarchical decomposition, opening up a new dimension by which they could take apart a problem and reason about its solution. In part, Polya's heuristics have seemed vague and too general, too hard to follow in real problems (Newell, 1983). But progress in AI programming, particularly expert system design, has suggested a vocabulary of structural concepts that we now see must be conveyed along with the heuristics to make them intelligible (see Chapter 29). Developing in parallel with Papert's educational experiments and capitalizing even more directly on AI technology, programs called intelligent tutoring systems (ITS) were constructed in the 1970s.

The success of MYCIN-like systems has demonstrated that for many diagnostic tasks expert behavior can be successfully captured in simple goaldirected production systems. However, even for this class of problems, difficulties have arisen with both the representation and control mechanisms. One such system, PUFF (Kunz et al., 1978), has established a creditable record in the domain of pulmonary function diagnosis. The representation problems in PUFF are manifest in a number of rules that have awkward premises and conclusions. The control problems are somewhat more severe. Physicians have criticized PUFF on the grounds that it asks questions that do not follow a logical line of reasoning and that it does not notice data that are atypical or erroneous for the determined diagnosis. In the CENTAUR sygtem, described in Chapter 23, an attempt was made to correct representational deficiencies by using prototypes (frames) to characterize some of the system's knowledge.

CENTAUR was designed in response to problems that occurred while using a purely rule-based system. The CENTAUR system offers an appropriate environment in which to experiment with knowledge representation issues such as determining what knowledge is most easily represented in rules and what is most easily represented in frames. In summary, much research remains to be done on this and associated knowledge representation issues. This present research is one attempt to make explicit the art of choosing the knowledge representation in AI by drawing comparisons between various approaches and by identifying the reasons for selecting one fundamental approach over another.

FIGURE 22-2 Summary of conclusions drawn by VM based on four hours of patient data. Current and optimal patient therapy stages are represented by their first letter: V -- VOLUME, A ASSIST, C controlled mandatory ventilation,/ changing. A double bar () is printed for each ten-minute interval in which the conclusion on the left is made.