Fine-Grained Neural Network Explanation by Identifying Input Features with Predictive Information Y ang Zhang

Neural Information Processing Systems 

One principal approach for illuminating a black-box neural network is feature attribution, i.e. identifying the importance of input features for the network's prediction.