Google Creates New SOTA Text-Image Generation Framework

Deep neural networks based on Generative Adversarial Networks (GANs) have enabled end-to-end trainable photorealistic text-to-image generation. Researchers have also developed methods designed to increase user control over the process, such as dialogue-based methods that enable inputting instructions to designate the relative positions of objects in a generated scene. However, the language that can be used in these processes is restricted, and the generated images are limited to synthetic 3D visualizations or cartoons. A team from Google Research has targeted these text-to-image shortcomings with a new system called Tag-Retrieve-Compose Synthesize (TReCS), which exploits both user text and mouse traces. The method is proposed in the recent paper Text-to-Image Generation Grounded by Fine-Grained User Attention.