Intention-aware policy graphs: answering what, how, and why in opaque agents

However, precisely because of the definition of such a task, the result is an artefact that, unless explicitly designed to be transparent, is often not interpretable or, hence, trustworthy (Zhang et al., 2021; Lipton, 2017). This is where the field of Explainable Artificial Intelligence (XAI) shines through. A model explanation is an exercise in communication between a sender or source (i.e. the model or one of its components) and a receiver (i.e. the explainee, a human or another processor for a downstream task) that describes the relevant context or the causes surrounding some facts (Lewis, 1986; Miller, 2019; Wright, 2004), which in the context of AI is often related to its final or intermediary outputs or decisions. Any such communicative act can be considered an explanation, but not all explanations may be useful or even desirable. According to empirical studies (Slugoski et al., 1993), it can be argued that the form of an explanation must depend on its function as an answer to a question within a conversational framework. Furthermore, in the words of Herbert Paul Grice (Grice, 1975), for a communicative act to be useful, four maxims should be followed: 1. Manner: the message or explanans should be comprehensible and clear to the receiver, which within the context of XAI is often referred to as interpretability (Lipton, 2017), 2. Quality: the message contains truthful information; in the context of XAI, reliability or explanation verification (Zhou et al., 2021b; Slack et al., 2021; Arias-Duart et al., 2022), 3. Quantity: the length of a message should be just enough to be informative, often a heuristic implicitly agreed upon in the design of explainable systems which depends on both the sender and the code it uses, and 4. Relation: the explanation should be relevant to the given context, significant when one can keep searching for causes of causes beyond the scope of relevance.