to

### Subsampling Generative Adversarial Networks: Density Ratio Estimation in Feature Space with Softplus Loss

Subsampling Generative Adversarial Networks: Density Ratio Estimation in Feature Space with Softplus Loss Xin Ding 1 1 University of British Columbia September 25, 2019 Abstract Filtering out unrealistic images from trained generative adversarial networks (GANs) starts attracting people's attention recently. Two density ratio based subsampling methods--Discriminator Rejection Sampling (DRS) and Metropolis-Hastings GAN (MH-GAN)--are recently proposed, and their effectiveness in improving GANs are demonstrated on multiple datasets. However, DRS and MH-GAN are developed based on discriminator based density ratio estimation (DRE) methods so they may not work well if the discriminator in the trained GAN is far away from its optimality. Moreover, they do not apply to some GANs (e.g., MMD-GAN). In this paper, we propose a novel Softplus (SP) loss for DRE based on which we develop a sample-based DRE method in a feature space learned by a specially designed and pre-trained ResNet-34 (DRE-F-SP). We derive the rate of convergence of a density ratio model trained under the SP loss. Then, we introduce three different density ratio based subsampling methods (DRE-F-SP RS, DRE-F-SP MH, and DRE-F-SP SIR) for GANs based on DRE-F-SP. Our subsampling methods do not rely on the optimality of the discriminator and are suitable for all types of GANs. We empirically show our subsampling approach can substantially outperform DRS and MH-GAN on a synthetic dataset and the CIF AR-10 dataset, using multiple GANs. 1 Introduction Generative adversarial networks (GANs) first introduced by [1] are well-known and powerful generative models for image synthesis and have been applied to various types of image-related tasks [2, 3, 4, 5, 6, 7]. The vanilla GANs proposed by [1] consist of two neural networks: a generator and a discriminator. The generator is trained to generate fake images to fool the discriminator while the discriminator is trained to distinguish fake images from real ones. To enhance the quality of fake images generated from the vanilla GANs, many subsequent works have been done on improving the training procedure of GANs, such as adopting large scale training schemes (e.g., BigGAN [8]), novel normalization methods (e.g., SN-GAN [9]), advanced GAN architectures (e.g., SAGAN [10]), and different loss functions (e.g., WGAN [11, 12] and MMD-GAN [13]). Instead of improving the training procedure, we are more interested in post-processing fake images from a trained GAN (i.e., subsampling fake images to filter out unrealistic images).

### Your GAN is Secretly an Energy-based Model and You Should use Discriminator Driven Latent Sampling

We show that the sum of the implicit generator log-density $\log p_g$ of a GAN with the logit score of the discriminator defines an energy function which yields the true data density when the generator is imperfect but the discriminator is optimal, thus making it possible to improve on the typical generator (with implicit density $p_g$). To make that practical, we show that sampling from this modified density can be achieved by sampling in latent space according to an energy-based model induced by the sum of the latent prior log-density and the discriminator output score. This can be achieved by running a Langevin MCMC in latent space and then applying the generator function, which we call Discriminator Driven Latent Sampling~(DDLS). We show that DDLS is highly efficient compared to previous methods which work in the high-dimensional pixel space and can be applied to improve on previously trained GANs of many types. We evaluate DDLS on both synthetic and real-world datasets qualitatively and quantitatively. On CIFAR-10, DDLS substantially improves the Inception Score of an off-the-shelf pre-trained SN-GAN~\citep{sngan} from $8.22$ to $9.09$ which is even comparable to the class-conditional BigGAN~\citep{biggan} model. This achieves a new state-of-the-art in unconditional image synthesis setting without introducing extra parameters or additional training.

### Discriminator Contrastive Divergence: Semi-Amortized Generative Modeling by Exploring Energy of the Discriminator

Generative Adversarial Networks (GANs) have shown great promise in modeling high dimensional data. The learning objective of GANs usually minimizes some measure discrepancy, \textit{e.g.}, $f$-divergence~($f$-GANs) or Integral Probability Metric~(Wasserstein GANs). With $f$-divergence as the objective function, the discriminator essentially estimates the density ratio, and the estimated ratio proves useful in further improving the sample quality of the generator. However, how to leverage the information contained in the discriminator of Wasserstein GANs (WGAN) is less explored. In this paper, we introduce the Discriminator Contrastive Divergence, which is well motivated by the property of WGAN's discriminator and the relationship between WGAN and energy-based model. Compared to standard GANs, where the generator is directly utilized to obtain new samples, our method proposes a semi-amortized generation procedure where the samples are produced with the generator's output as an initial state. Then several steps of Langevin dynamics are conducted using the gradient of the discriminator. We demonstrate the benefits of significant improved generation on both synthetic data and several real-world image generation benchmarks.

### Discriminator optimal transport

Within a broad class of generative adversarial networks, we show that discriminator optimization process increases a lower bound of the dual cost function for the Wasserstein distance between the target distribution $p$ and the generator distribution $p_G$. It implies that the trained discriminator can approximate optimal transport (OT) from $p_G$ to $p$.Based on some experiments and a bit of OT theory, we propose a discriminator optimal transport (DOT) scheme to improve generated images. We show that it improves inception score and FID calculated by un-conditional GAN trained by CIFAR-10, STL-10 and a public pre-trained model of conditional GAN by ImageNet.

### Variational Approaches for Auto-Encoding Generative Adversarial Networks

Auto-encoding generative adversarial networks (GANs) combine the standard GAN algorithm, which discriminates between real and model-generated data, with a reconstruction loss given by an auto-encoder. Such models aim to prevent mode collapse in the learned generative model by ensuring that it is grounded in all the available training data. In this paper, we develop a principle upon which auto-encoders can be combined with generative adversarial networks by exploiting the hierarchical structure of the generative model. The underlying principle shows that variational inference can be used a basic tool for learning, but with the in- tractable likelihood replaced by a synthetic likelihood, and the unknown posterior distribution replaced by an implicit distribution; both synthetic likelihoods and implicit posterior distributions can be learned using discriminators. This allows us to develop a natural fusion of variational auto-encoders and generative adversarial networks, combining the best of both these methods. We describe a unified objective for optimization, discuss the constraints needed to guide learning, connect to the wide range of existing work, and use a battery of tests to systematically and quantitatively assess the performance of our method.