We develop Neuron Shapley as a new framework to quantify the contribution of individual neurons to the prediction and performance of a deep network.

Weaknesses: The idea of applying Shapley values for the understanding of deep neural networks is not new. Several works, such as Lundberg et al., 2017, have already discussed the theoretical motivation for using Shapley values as an attribution method to rank the importance of the input features. Lundberg et al., 2017 also proposed approximations like KernelSHAP and DeepSHAP, which are not compared to TMAB-Shapley. Besides this line of works, the idea of using Shapley values to rank the internal neurons has been proposed by the Stier et al., 2018 (cited) and Florin Leon, 2014 (not cited) in the context of pruning. Finally, Ancona et al., 2019 (not cited) proposed an approximation technique for Shapley values tailored for deep neural networks.

We develop Neuron Shapley as a new framework to quantify the contribution of individual neurons to the prediction and performance of a deep network. By accounting for interactions across neurons, Neuron Shapley is more effective in identifying important filters compared to common approaches based on activation patterns. Interestingly, removing just 30 filters with the highest Shapley scores effectively destroys the prediction accuracy of Inception-v3 on ImageNet. Visualization of these few critical filters provides insights into how the network functions. Neuron Shapley is a flexible framework and can be applied to identify responsible neurons in many tasks. We illustrate additional applications of identifying filters that are responsible for biased prediction in facial recognition and filters that are vulnerable to adversarial attacks. Removing these filters is a quick way to repair models. Enabling all these applications is a new multi-arm bandit algorithm that we developed to efficiently estimate Neuron Shapley values.