Consistency Regularization for Generative Adversarial Networks

A BSTRACT Generative Adversarial Networks (GANs) are known to be difficult to train, despite considerable research effort. Several regularization techniques for stabilizing training have been proposed, but they introduce nontrivial computational overheads and interact poorly with existing techniques like spectral normalization. In this work, we propose a simple, effective training stabilizer based on the notion of consistency regularization--a popular technique in the semi-supervised learning literature. In particular, we augment data passing into the GAN discriminator and penalize the sensitivity of the discriminator to these augmentations. We conduct a series of experiments to demonstrate that consistency regularization works effectively with spectral normalization and various GAN architectures, loss functions and optimizer settings. Our method achieves the best FID scores for unconditional image generation compared to other regularization methods on CIFAR-10 and CelebA. In the original setting, GANs are composed of two neural networks trained with competing goals: the generator is trained to synthesize realistic samples to fool the discriminator and the discriminator is trained to distinguish real samples from fake ones produced by the generator. One major problem with GANs is the instability of the training procedure and the general sensitivity of the results to various hyperparameters (Salimans et al., 2016).