Explaining a Series of Models by Propagating Shapley Values

With the widespread adoption of machine learning (ML), series of models (i.e., where the outputs of predictive models are used as inputs to separate predictive models) are increasingly common. Examples include: (1) stacked generalization, a widely used technique [1-5] to improve generalization performance by ensembling the predictions of many models (called base-learners) using another model (called a meta-learner) [6], (2) neural network feature extraction, where models are trained on features extracted using neural networks [7, 8], typically for structured data [9-11], and (3) consumer scores, where predictive models that describe a specific behavior (e.g., credit scores [12]) are used as inputs to downstream predictive models. For example, a bank may use a model to predict customers' loan eligibility on the basis of their bank statements and their credit score, which itself is often a predictive model [13]. Explaining a series of models is crucial for debugging and building trust, even more so because a series of models is inherently harder to explain compared to a single model. One popular paradigm for explaining models are local feature attributions, which explain why a model makes a prediction for a single sample (known as the "explicand" [14]). Existing model-agnostic local feature attribution methods (e.g., IME [15], LIME [16], KernelSHAP [17]) work regardless of the specific model being explained. They can explain a series of models, but suffer from two distinct shortcomings: (1) their sampling-based estimates of feature importance are inherently variable, and (2) they have high computational cost which may not be tractable for large pipelines.