In light of the recent widespread adoption of AI systems, understanding the internal information processing of neural networks has become increasingly critical.

Recall measures the fraction ofcommon samples that are retrievedascorrect common class, while specificity measures thefraction ofprivatesamples thatarenotretrieved. Fig. S1(b) shows the sensitivity ofγ, where γ is the rough boundary for splitting positive and negative in adaptive filling. For the cosine similarity of two ℓ2-normalized features, the similarity value is limited from 1to1, where higher value indicates higher similarity. Suchself-supervisedlearning methods encourage the consistency between two augmentations of one image. The display images for source prototypes are chosen by finding the nearest source instance of the prototype.

Tiny-ImageNet isasmall subset of ImageNet dataset, containing 100,000 training images, 10,000 validation images, and 10,000 testing images separated in 200 different classes, dimensionsofwhichare64 64pixels. Here,anapproximate featureprobability q(Z) is introduced to approximate the true feature probabilityp(Z). The additional results are illustrated in Figure 1. We provide additional feature visualization under various adversarial attack methods including NRF in Figure 1-5 (CIFAR-10, SVHN, and Tiny-ImageNet are utilized). Moreover,thedistilled features still include therobustand brittle information eveninthefailed attack examples.

A precise understanding of why units in an artificial network respond to certain stimuli would constitute a big step towards explainable artificial intelligence. One widely used approach towards this goal is to visualize unit responses via activation maximization. These feature visualizations are purported to provide humans with precise information about the image features that cause a unit to be activated - an advantage over other alternatives like strongly activating dataset samples. If humans indeed gain causal insight from visualizations, this should enable them to predict the effect of an intervention, such as how occluding a certain patch of the image (say, a dog's head) changes a unit's activation. Here, we test this hypothesis by asking humans to decide which of two square occlusions causes a larger change to a unit's activation.Both a large-scale crowdsourced experiment and measurements with experts show that on average the extremely activating feature visualizations by Olah et al. (2017) indeed help humans on this task ($68 \pm 4$% accuracy; baseline performance without any visualizations is $60 \pm 3$%). However, they do not provide any substantial advantage over other visualizations (such as e.g.

In light of the recent widespread adoption of AI systems, understanding the internal information processing of neural networks has become increasingly critical.