Towards Human-Centered Explainable AI: the journey so far

"So, the machine has high accuracy and explains its decisions, but we still don't have engagement with our users?" I asked seeking clarification on a rather perplexing situation. Aware of my prior work in Explainable AI (XAI) around rationale generation, a prominent tech company had just hired me to solve a unique problem. They invested significant resources to build an AI-powered cybersecurity system that aims to help analysts manage firewall configurations, especially "bloat" that happens when people forget to close open ports. Over time, these open ports accumulate and create security vulnerability. Not only did this system have commendable accuracy, it also tried to explain its decision via technical (or algorithmic) transparency. But, there was almost zero to no traction amongst its users. I think we just need better models…we need to build better rationales [natural language explanations] … guess that's why we brought you in!" the team's director chuckled as we continued the ...