Researchers at the Korea Advanced Institute of Science and Technology and Pohang University of Science and Technology have introduced a machine learning algorithm system, InstaGAN, which can perform multiple instance-aware image-to-image translation tasks -- such as replacing sheep in photos with giraffes -- on multiple image datasets. The paper InstaGAN: Instance-Aware Image-to-Image Translation has been accepted by the respected International Conference on Learning Representations (ICLR) 2019, which will take place this May in New Orleans, USA. An image-to-image translation system is a system that learns to map an input image onto an output image. Unsupervised image-to-image translation has garnered considerable research attention recently in part due to the rapid development of generative adversarial networks (GANs) that now power the technique. Previous methods were not suitable for challenging tasks, for example if the image has multiple target instances or if the translation task involves challenging shapes.

It might sound far-fetched, but those are just a couple of the feats a machine learning algorithm designed by researchers at the Korea Advanced Institute of Science and Technology and the Pohang University of Science and Technology can accomplish after ingesting large datasets of images. It's described in a new paper ("InstaGAN: Instance-Aware Image-to-Image Translation") published on the preprint server Arxiv.org this week. Image-to-image translation systems -- that is, systems that learn the mapping from input image to output image -- aren't anything new, to be clear. Only earlier this month, Google AI researchers developed a model that can realistically insert an object in a photo by predicting its scale, occlusions, pose, shape, and more. But as the creators of InstaGAN wrote in the paper, even state-of-the-art methods aren't perfect.

Unsupervised image-to-image translation has gained considerable attention due to the recent impressive progress based on generative adversarial networks (GANs). However, previous methods often fail in challenging cases, in particular, when an image has multiple target instances and a translation task involves significant changes in shape, e.g., translating pants to skirts in fashion images. To tackle the issues, we propose a novel method, coined instance-aware GAN (InstaGAN), that incorporates the instance information (e.g., object segmentation masks) and improves multi-instance transfiguration. The proposed method translates both an image and the corresponding set of instance attributes while maintaining the permutation invariance property of the instances. To this end, we introduce a context preserving loss that encourages the network to learn the identity function outside of target instances. We also propose a sequential mini-batch inference/training technique that handles multiple instances with a limited GPU memory and enhances the network to generalize better for multiple instances. Our comparative evaluation demonstrates the effectiveness of the proposed method on different image datasets, in particular, in the aforementioned challenging cases. Code and results are available in https://github.com/sangwoomo/instagan