Core to human-centered AI is explainability. If a machine cannot explain its reasoning in a way that humans understand and on human terms, the AI isn't working for people. Researchers from Georgia Institute of Technology, Cornell University and the University of Kentucky recently published the results of teaching a machine to generate conversational explanations of its model's internal state and action data representations in real-time. They tested whether people like the machine to tell them how it made decisions, and what characteristics of explanations drove people's perceptions of explainability. Relatability is key to understandability – when an AI uses natural language to explain itself, people put themselves in the AI's shoes and evaluate understandability based on whether the AI gives the same reasons they would.

Automated rationale generation is an approach for real-time explanation generation whereby a computational model learns to translate an autonomous agent's internal state and action data representations into natural language. Training on human explanation data can enable agents to learn to generate human-like explanations for their behavior. In this paper, using the context of an agent that plays Frogger, we describe (a) how to collect a corpus of explanations, (b) how to train a neural rationale generator to produce different styles of rationales, and (c) how people perceive these rationales. We conducted two user studies. The first study establishes the plausibility of each type of generated rationale and situates their user perceptions along the dimensions of confidence, humanlike-ness, adequate justification, and understandability. The second study further explores user preferences between the generated rationales with regard to confidence in the autonomous agent, communicating failure and unexpected behavior. Overall, we find alignment between the intended differences in features of the generated rationales and the perceived differences by users. Moreover, context permitting, participants preferred detailed rationales to form a stable mental model of the agent's behavior.