Appendix

Despite initial evidence that explanations might be useful for detecting that a model is reliant on spurious signals [Lapuschkin et al., 2019, Rieger et al., 2020], a different line of work directly counters this evidence. Zimmermann et al. [2021] showed that feature visualizations [Olah et al., 2017] are not more effective than dataset examples at improving a human's understanding of the features that highly activate a DNN's intermediate neuron. Increasing evidence demonstrates that current post hoc explanation approaches might be ineffective for model debugging in practice [Chen et al., 2021, Alqaraawi et al., 2020, Ghassemi et al., 2021, Balagopalan et al., 2022, Poursabzi-Sangdeh et al., 2018, Bolukbasi et al., 2021]. In a promising demonstration, Lapuschkin et al. [2019] apply a clustering procedure to the LRP saliency masks derived from a trained model. In the application, the clusters that emerge are able to separate groups of inputs where, presumably, the model relies on different features for its output decision. This work differs from that in a key way: Lapuschkin et al. [2019] demonstration is to seek understanding of the model behavior and not to perform slice discovery. There is no reason why a low performing cluster should emerge from such clustering procedure. Schioppa et al. [2022] address this problem by forming a low-rank approximation of H They choose D to be around 50 in their experiments.