Natural Language Interaction with Explainable AI Models

This paper presents an explainable AI (XAI) system that provides explanations for its predictions. The system consists of two key components - namely, the prediction And-Or graph (AOG) model for recognizing and localizing concepts of interest in input data, and the XAI model for providing explanations to the user about the AOG's predictions. In this work, we Figure 1: Two frames (scenes) of a video: (a) focus on the XAI model specified to interact top-left image (scene1) shows two persons sitting with the user in natural language, at the reception and others entering the auditorium whereas the AOG's predictions are considered and (b) top-right (scene2) image people running given and represented by the corresponding out of an auditorium. Bottom-left shows the parse graphs (pg's) of the AOG. AOG parse graph (pg) for the top-left image and Our XAI model takes pg's as input and Bottom-right shows the pg for the top-right image provides answers to the user's questions using the following types of reasoning: direct evidence (e.g., detection scores), Consider for example, two frames (scenes) of part-based inference (e.g., detected parts a video shown in Figure 1. An action detection provide evidence for the concept asked), model might predict that two people in the scene1 and other evidences from spatiotemporal are in sitting posture. User might be interested context (e.g., constraints from the spatiotemporal to know more details about the prediction such surround). We identify several as: Why do the model think the people are in sitting correlations between user's questions posture? Why not standing instead of sitting? and the XAI answers using Youtube Action Why two persons are sitting instead of one?