This article gives an overview of two recent events on the validation and verification of knowledge-based systems: (1) the 1997 European Symposium on the Verification and Validation of Knowledge-Based Systems (EUROVAV-97) and (2) the Four-teenth National Conference on Artificial Intelligence Workshop on the Verification and Validation of Knowledge- Based Systems. To give an integrated view of current research issues in this field, we organized this article along thematic lines, unifying the reports of the two separate meetings. Our report focuses on the trends that we think will be important in the near future in this field.

To remain competitive, enterprises must become increasingly agile and integrated across their functions. Enterprise models play a critical role in this integration, enabling better designs for enterprises, analysis of their performance, and management of their operations. This article motivates the need for enterprise models and introduces the concepts of generic and deductive enterprise models. It reviews research to date on enterprise modeling and considers in detail the Toronto virtual enterprise effort at the University of Toronto.

In the last few years, we have witnessed a major growth in the use of empirical methods in AI. In part, this growth has arisen from the availability of fast networked computers that allow certain problems of a practical size to be tackled for the first time. There is also a growing realization that results obtained empirically are no less valuable than theoretical results. I identify some of the emerging trends in this area by describing a recent workshop that brought together researchers using empirical methods as far apart as robotics and knowledge-based systems.

In the last few years, we have witnessed a major growth in the use of empirical methods in AI. In part, this growth has arisen from the availability of fast networked computers that allow certain problems of a practical size to be tackled for the first time. There is also a growing realization that results obtained empirically are no less valuable than theoretical results. Experiments can, for example, offer solutions to problems that have defeated a theoretical attack and provide insights that are not possible from a purely theoretical analysis. I identify some of the emerging trends in this area by describing a recent workshop that brought together researchers using empirical methods as far apart as robotics and knowledge-based systems.

The emerging literature on combined systems is directed at domains where the prediction of human behavior is not required. Professionals concerned with human outcomes make decisions that are intuitive or analytic or some combination of both. Justifications and methodology are presented for combining analytic and intuitive agents in an expert system that supports professional decision making. The system presented demonstrates the challenges and opportunities inherent in developing and using AI-collaborative technology to solve social problems.

CHEMREG is a large knowledge-based system used by Air Products and Chemicals, Inc., to support compliance with regulatory requirements for communicating health and safety information in the shipping and handling of chemical products. This article concentrates on one of the knowledge bases in this system: the case-based reasoner. The case-based reasoner addresses the issue of how proper communication of public health and safety information can be ensured while rapid and cost-effective product evaluation is allowed in the absence of actual hazard testing of the product. CHEMREG generates estimates of hazard data for new products from similar products using an existing relational database as a case library. Implementation of the case-based reasoner in rules and objects using a commercial knowledge-based system shell is described. Although some refinements remain, the performance of the case-based reasoner has met its design goals.

AI theory and its technology is rarely consulted in attempted resolutions of social problems. Solutions often require that decision-analytic techniques be combined with expert systems. The emerging literature on combined systems is directed at domains where the prediction of human behavior is not required. A foundational shift in AI presuppositions to intelligent agents working in collaboration provides an opportunity to explore efforts to improve the performance of social institutions that depend on accurate prediction of human behavior. Professionals concerned with human outcomes make decisions that are intuitive or analytic or some combination of both. The relative efficacy of each decision type is described. Justifications and methodology are presented for combining analytic and intuitive agents in an expert system that supports professional decision making. Psychological grounds for the allocation of functions to agents are reviewed. Jury selection, the prototype domain, is described as a process typical of others that, at their core, require the prediction of human behavior. The domain is used to demonstrate the formal components, steps in construction, and challenges of developing and testing a hybrid system based on the allocation of function. The principle that the research taught us about the allocation of function is "the rational and predictive primacy of a statistical agent to an intuitive agent in construction of a production system." We learned that the reverse of this principle is appropriate for identifying and classifying human responses to questions and generally dealing with unexpected events in a courtroom and elsewhere. This principle and approach should be paradigmatic of the class of collaborative models that capitalizes on the unique strengths of AI knowledge-based systems. The methodology used in the courtroom is described along with the history of the project and implications for the development of related AI systems. Empirical data are reported that portend the possibility of impressive predictive ability in the combined approach relative to other current approaches. Problems encountered and those remaining are discussed, including the limits of empirical research and standards of validation. The system presented demonstrates the challenges and opportunities inherent in developing and using AI-collaborative technology to solve social problems.

MetLife processes over 260,000 life insurance applications a year. Underwriting of these applications is labor intensive. Automation is difficult because the applications include many free-form text fields. MetLife's intelligent text analyzer (MITA) uses the information-extraction technique of natural language processing to structure the extensive textual fields on a life insurance application. Knowledge engineering, with the help of underwriters as domain experts, was performed to elicit significant concepts for both medical and occupational textual fields. A corpus of 20,000 life insurance applications provided the syntactical and semantic patterns in which these underwriting concepts occur. These patterns, in conjunction with the concepts, formed the frameworks for information extraction. Extension of the information-extraction work developed by Wendy Lehnert was used to populate these frameworks with classes obtained from the systematized nomenclature of human and veterinary medicine and the Dictionary of Occupational Titles ontologies. These structured frameworks can then be analyzed by conventional knowledge-based systems. MITA is currently processing 20,000 life insurance applications a month. Eighty-nine percent of the textual fields processed by MITA exceed the established confidence-level threshold and are potentially available for further analysis by domain-specific analyzers.

This article reports on the development and implementation of DISXPERT, an intelligent rule-based system tool for referral of social security disability recipients to vocational rehabilitation services. The growing use of paraprofessionals as caseworkers responsible for assessment in the social services area provides fertile domain areas for new and innovative application of intelligent system technology. The main function of DISXPERT is to provide support to paraprofessional caseworkers in reaching unbiased and consistent assessment decisions regarding referral of clients to vocational rehabilitation services. The results after four years of use demonstrate that paraprofessionals using DISXPERT can make assessments in less time and with a level of accuracy superior to the vocational rehabilitation domain professionals using manual methods. This article discusses the problem domain, the design and development of the system, uses of AI technology, payoffs, and deployment and maintenance of the system.

This historical, developmental approach to AI was visible throughout the conference. The presented at the 1997 Mobile Robot the end of the twentieth century," strong tutorial sessions, workshops, Competition, for example, time was "A computer beat natural language, machine learning to Robot Competition get better, and the the world champion at chess. A robot reasoning and representation, the events get harder. This year, the events is on Mars making a few of its own decisions." More than 20 teams competed AI's visible achievements, noted Allen deftly compared AI to the stages this year. At its heart, it is a technical of human development.