Explaining the predictions of AI models is paramount in safety-critical applications, such as in legal or medical domains. One form of explanation for a prediction is an extractive rationale, i.e., a subset of features of an instance that lead the model to give its prediction on that instance. For example, the subphrase ``he stole the mobile phone'' can be an extractive rationale for the prediction of ``Theft''. Previous works on generating extractive rationales usually employ a two-phase model: a selector that selects the most important features (i.e., the rationale) followed by a predictor that makes the prediction based exclusively on the selected features. One disadvantage of these works is that the main signal for learning to select features comes from the comparison of the answers given by the predictor to the ground-truth answers. In this work, we propose to squeeze more information from the predictor via an information calibration method. More precisely, we train two models jointly: one is a typical neural model that solves the task at hand in an accurate but black-box manner, and the other is a selector-predictor model that additionally produces a rationale for its prediction. The first model is used as a guide for the second model. We use an adversarial technique to calibrate the information extracted by the two models such that the difference between them is an indicator of the missed or over-selected features. In addition, for natural language tasks, we propose a language-model-based regularizer to encourage the extraction of fluent rationales. Experimental results on a sentiment analysis task, a hate speech recognition task as well as on three tasks from the legal domain show the effectiveness of our approach to rationale extraction.

AI is fast becoming embedded in industries, economies and lives, making decisions, recommendations and predictions. These trends mean it's business-critical to understand how AI-enabled systems arrive at specific outputs. It's not enough for an AI algorithm to generate the right result--knowing "the reason why" is now a business fundamental. The process has to be transparent, trustworthy and compliant--far removed from the opaque "black-box" concept that has characterized some AI advances in recent times. At the same time, these advances should not be stifled.

The XAI literature is decentralized, both in terminology and in publication venues, but recent years saw the community converge around keywords that make it possible to more reliably discover papers automatically. We use keyword search using the SemanticScholar API and manual curation to collect a well-formatted and reasonably comprehensive set of 5199 XAI papers, available at https://github.com/alonjacovi/XAI-Scholar . We use this collection to clarify and visualize trends about the size and scope of the literature, citation trends, cross-field trends, and collaboration trends. Overall, XAI is becoming increasingly multidisciplinary, with relative growth in papers belonging to increasingly diverse (non-CS) scientific fields, increasing cross-field collaborative authorship, increasing cross-field citation activity. The collection can additionally be used as a paper discovery engine, by retrieving XAI literature which is cited according to specific constraints (for example, papers that are influential outside of their field, or influential to non-XAI research).

Interactive Artificial Intelligence (AI) agents are becoming increasingly prevalent in society. However, application of such systems without understanding them can be problematic. Black-box AI systems can lead to liability and accountability issues when they produce an incorrect decision. Explainable AI (XAI) seeks to bridge the knowledge gap, between developers and end-users, by offering insights into how an AI algorithm functions. Many modern algorithms focus on making the AI model "transparent", i.e. unveil the inherent functionality of the agent in a simpler format. However, these approaches do not cater to end-users of these systems, as users may not possess the requisite knowledge to understand these explanations in a reasonable amount of time. Therefore, to be able to develop suitable XAI methods, we need to understand the factors which influence subjective perception and objective usability. In this paper, we present a novel user-study which studies four differing XAI modalities commonly employed in prior work for explaining AI behavior, i.e. Decision Trees, Text, Programs. We study these XAI modalities in the context of explaining the actions of a self-driving car on a highway, as driving is an easily understandable real-world task and self-driving cars is a keen area of interest within the AI community. Our findings highlight internal consistency issues wherein participants perceived language explanations to be significantly more usable, however participants were better able to objectively understand the decision making process of the car through a decision tree explanation. Our work also provides further evidence of importance of integrating user-specific and situational criteria into the design of XAI systems. Our findings show that factors such as computer science experience, and watching the car succeed or fail can impact the perception and usefulness of the explanation.

Counterfactual explanations (CEs) are methods for generating an alternative scenario that produces a different desirable outcome. For example, if a student is predicted to fail a course, then counterfactual explanations can provide the student with alternate ways so that they would be predicted to pass. The applications are many. However, CEs are currently generated from machine learning models that do not necessarily take into account the true causal structure in the data. By doing this, bias can be introduced into the CE quantities. I propose in this study to test the CEs using Judea Pearl's method of computing counterfactuals which has thus far, surprisingly, not been seen in the counterfactual explanation (CE) literature. I furthermore evaluate these CEs on three different causal structures to show how the true underlying causal structure affects the CEs that are generated. This study presented a method of evaluating CEs using Pearl's method and it showed, (although using a limited sample size), that thirty percent of the CEs conflicted with those computed by Pearl's method. This shows that we cannot simply trust CEs and it is vital for us to know the true causal structure before we blindly compute counterfactuals using the original machine learning model.

This paper presented an XAI system specification and an interactive dialogue model to facilitate the creation of Explanation Experiences (EE). Such specifications combine the knowledge of XAI, domain and system experts of a use case to formalise target user groups and their explanation needs and to implement explanation strategies to address those needs. Formalising the XAI system promotes the reuse of existing explainers and known explanation needs that can be refined and evolved over time using user evaluation feedback. The abstract EE dialogue model formalised the interactions between a user and an XAI system. The resulting EE conversational chatbot is personalised to an XAI system at run-time using the knowledge captured in its XAI system specification. This seamless integration is enabled by using Behaviour Trees (BT) to conceptualise both the EE dialogue model and the explanation strategies. In the evaluation, we discussed several desirable properties of using BTs over traditionally used STMs or FSMs. BTs promote the reusability of dialogue components through the hierarchical nature of the design. Sub-trees are modular, i.e. a sub-tree is responsible for a specific behaviour, which can be designed in different levels of granularity to improve human interpretability. The EE dialogue model consists of abstract behaviours needed to capture EE, accordingly, it can be implemented as a conversational, graphical or text-based interface which caters to different domains and users. There is a significant computational cost when using BTs for modelling dialogue, which we mitigate by using memory. Overall, we find that the ability to create robust conversational pathways dynamically makes BTs a good candidate for designing and implementing conversation for creating explanation experiences.

In their project report paper Do We Need Explainable AI in Companies? Investigation of Challenges, Expectations, and Chances from Employees' Perspective, Katharina Weitz, Chi Tai Dang and Elisabeth André investigated employees' specific needs and attitudes towards AI. In this interview, Katharina and Chi Tai tell us more about this work. Our paper examines the current state of AI use in companies. It is particularly important to us to capture the perspective of employees.

The success of neural networks builds to a large extent on their ability to create internal knowledge representations from real-world high-dimensional data, such as images, sound, or text. Approaches to extract and present these representations, in order to explain the neural network's decisions, is an active and multifaceted research field. To gain a deeper understanding of a central aspect of this field, we have performed a targeted review focusing on research that aims to associate internal representations with human understandable concepts. In doing this, we added a perspective on the existing research by using primarily deductive nomological explanations as a proposed taxonomy. We find this taxonomy and theories of causality, useful for understanding what can be expected, and not expected, from neural network explanations. The analysis additionally uncovers an ambiguity in the reviewed literature related to the goal of model explainability; is it understanding the ML model or, is it actionable explanations useful in the deployment domain?

Banks hold a societal responsibility and regulatory requirements to mitigate the risk of financial crimes. Risk mitigation primarily happens through monitoring customer activity through Transaction Monitoring (TM). Recently, Machine Learning (ML) has been proposed to identify suspicious customer behavior, which raises complex socio-technical implications around trust and explainability of ML models and their outputs. However, little research is available due to its sensitivity. We aim to fill this gap by presenting empirical research exploring how ML supported automation and augmentation affects the TM process and stakeholders' requirements for building eXplainable Artificial Intelligence (xAI). Our study finds that xAI requirements depend on the liable party in the TM process which changes depending on augmentation or automation of TM. Context-relatable explanations can provide much-needed support for auditing and may diminish bias in the investigator's judgement. These results suggest a use case-specific approach for xAI to adequately foster the adoption of ML in TM.

If you've ever gotten a letter from a bank that explained how different financial issues influenced a credit application, you've seen explainable AI at work -- a computer used math and a set of complex formulas to calculate a score and determine whether to approve or deny your application. In making that decision, some data points were either more or less important. Maybe your long history of on-time payments or your low amount of debt contributed to your application's approval. Similarly, explainable AI shows humans how it arrived at a decision by evaluating different inputs in its calculations. While that might sound obscure or only relevant to the most hardcore data people, explainable AI brings significant business advantages that anyone interested in applying AI should consider.