Learning to Sketch with Shortcut Cycle Consistency – Arxiv Vanity

In order to achieve photo-to-sketch synthesis with noisy photo-sketch pairs as supervision, we address the limitations of existing cross-domain image translation models by proposing a novel framework based on multi-task supervised and unsupervised hybrid learning (see Figure 2(c)). Taking an encoder-decoder architecture, our primary task is D(E(photo)) sketch) where a photo is first encoded by E and then decoded into a sketch by D. To help learn a better encoder and decoder, we introduce the inverse problem (D(E(sketch)) photo) so that the supervised model learning can be done in both directions. Importantly, we also introduce two unsupervised learning tasks for within-domain reconstruction, \ie, D(E(photo)) photo and D(E(sketch)) sketch. This hybrid learning framework differs significantly from existing approaches in that: (1) It combines supervised and unsupervised learning in a multi-task learning framework in order to make the best use of the noisy supervision signal. In particular, by sharing the encoder and decoder in various tasks, a more robust and effective encoder and decoder for the main photo-to-sketch synthesis task can be obtained.