Visualizing and Understanding Generative Adversarial Networks (Extended Abstract)

The ability of generative adversarial networks to render nearly photorealistic images leads us to ask: What does a GAN know? For example, when a GAN generates a door on a building but not in a tree (Figure 1a), we wish to understand whether such structure emerges as pure pixel patterns without explicitrepresentation, or if the GAN contains internal variables that correspond to human-perceived objects such as doors, buildings, and trees. And when a GAN generates an unrealistic image (Figure 1f), we want to know if the mistake is caused by specific variables in the network. We present a method for visualizing and understanding GANs at different levels of abstraction, from each neuron, to each object, to the relationship between different objects. Beginning witha Progressive GAN (Karras et al., 2018) trained to generate scenes (Figure 1b), we first identify a group of interpretable units that are related to semantic classes (Figure 1a,Figure 2). These units' featuremaps closely match the semantic segmentation of a particular object class (e.g., doors). Then, we directly intervene within the network to identify sets of units that cause a type of object to disappear (Figure1c) or appear (Figure 1d). Finally, we study contextual relationships by observing where we can insert the object concepts in new images and how this intervention interacts with other objects in the image (Figure 1d, Figure 8). This framework enables several applications: comparing internal representationsacross different layers, GAN variants, and datasets (Figure 2); debugging and improving GANs by locating and ablating artifact-causing units (Figure 1e,f,g); understanding contextual relationships between objects in natural scenes (Figure 8,Figure 9); and manipulating images with interactive object-level control (video).