Nothing Else Matters: Model-Agnostic Explanations By Identifying Prediction Invariance

At the core of interpretable machine learning is the question of whether humans are able to make accurate predictions about a model's behavior. Assumed in this question are three properties of the interpretable output: coverage, precision, and effort. Coverage refers to how often humans think they can predict the model's behavior, precision to how accurate humans are in those predictions, and effort is either the upfront effort required in interpreting the model, or the effort required to make predictions about a model's behavior. One approach to interpretable machine learning is designing inherently interpretable models. Visualizations of these models usually have perfect coverage, but there is a tradeoff between the accuracy of the model and the effort required to comprehend it - especially in complex domains like text and images, where the input space is very large, and accuracy is usually sacrificed for models that are compact enough to be comprehensible by humans. Experiments usually involve showing humans these visualizations, and measuring human precision when predicting the model's behavior on random instances, and the time (effort) required to make those predictions [7, 8, 9]. Model-agnostic explanations [12] avoid the need to trade off accuracy by treating the model as a black box. Explanations such as sparse linear models [11] (henceforth called linear LIME) or gradients [2, 10] can still exhibit high precision and low effort (which are de-facto requirements, as there is little point in explaining a model if explanations lead to poor understanding or are too complex) even for very complex models by providing explanations that are local in their scope (i.e.