A Sim2Real Approach for Identifying Task-Relevant Properties in Interpretable Machine Learning

In the context of human+AI interaction, explanations of the underlying function can provide additional information to assist the human in performing their task. Recent literature suggests that explanations with different properties are useful for different tasks [Liao et al., 2022, Lai et al., 2023, Chen et al., 2023, Jesus et al., 2021, Wang et al., 2019, Liao et al., 2020, Lim and Dey, 2009]. For example, in an AI-auditing task, the user may need to check whether the AI inappropriately relied on a forbidden feature, such as using gender in computing a credit score [Kaur et al., 2020, Hase and Bansal, 2020a, Lakkaraju et al., 2019]. In this case, we would want explanations that are faithful; that is, they reliably capture the underlying behavior of the function. On the other hand, suppose our goal is to help a user quickly understand the process by which a function produces its output; we can quantify the user's understanding by measuring the user's ability to approximate the function's output, given the input and an explanation [Hase and Bansal, 2020b, Chandrasekaran et al., 2018]. In this case, we may want explanations with low complexity, so that the user can effectively reason using the explanation in a limited amount of time.