A Formal Approach to Explainability

We regard explanations as a blending of the input sample and the model's output and offer a few definitions that capture various desired properties of the function that generates t hese explanations. We study the links between these properties a nd between explanation-generating functions and intermedia te representations of learned models and are able to show, for example, that if the activations of a given layer are consist ent with an explanation, then so do all other subsequent layers. In addition, we study the intersection and union of explanatio ns as a way to construct new explanations.