We now consider the gradient of the log-variance loss. Using the definition from Eq. 5, we see that From [Reiss, 2012, Lemma A.3.5] we have the bound Combining these estimates we arrive at the claimed result. The claim follows by direct calculation. In fact, it is possible to take K = 15 . Lemma 3 shows that the kurtosis term in our bound Eq. 16 can be bounded for Gaussian families.

The problem of explaining deep learning models, and model predictions generally, has attracted intensive interest recently. Many successful approaches forgo global approximations in order to provide more faithful local interpretations of the model's behavior. LIME develops multiple interpretable models, each approximating a large neural network on a small region of the data manifold and SP-LIME aggregates the local models to form a global interpretation. Extending this line of research, we propose a simple yet effective method, Norm-LIME for aggregating local models into global and class-specific interpretations. A human user study strongly favored class-specific interpretations created by NormLIME to other feature importance metrics. Numerical experiments confirm that NormLIME is effective at recognizing important features. Introduction As the applications of deep neural networks continue to expand, the intrinsic black-box nature of neural networks creates a potential trust issue. For application domains with high cost of prediction error, such as healthcare (Phan et al. 2017), it is necessary that human users can verify that a model learns reasonable representation of data and the rationale for its decisions are justifiable according to societal norms (Koh and Liang 2017; Fong and V edaldi 2018; Zhou et al. 2018; Lipton 2016; Langley 2019). An interpretable model, such as a linear sparse regression, lends itself readily to model explanation.

SmoothGrad and VarGrad are techniques that enhance the empirical quality of standard saliency maps by adding noise to input. However, there were few works that provide a rigorous theoretical interpretation of those methods. We analytically formalize the result of these noise-adding methods. As a result, we observe two interesting results from the existing noise-adding methods. First, SmoothGrad does not make the gradient of the score function smooth. Second, VarGrad is independent of the gradient of the score function. We believe that our findings provide a clue to reveal the relationship between local explanation methods of deep neural networks and higher-order partial derivatives of the score function.