Generating Sentences Using a Dynamic Canvas

Harshil Shah University College London Bowen Zheng University College London David Barber University College London & Alan Turing Institute Abstract We introduce the A ttentive Unsupervised T ext (W) riter (AUTR), which is a word level generative model for natural language. It uses a recurrent neural network with a dynamic attention and canvas memory mechanism to iteratively construct sentences. By viewing the state of the memory at intermediate stages and where the model is placing its attention, we gain insight into how it constructs sentences. We demonstrate that AUTR learns a meaningful latent representation for each sentence, and achieves competitive log-likelihood lower bounds whilst being computationally efficient. It is effective at generating and reconstructing sentences, as well as imputing missing words. 1 Introduction Latent variable models have recently enjoyed significant success when modelling images (Gregor et al. 2015; Rezende et al. 2016; Gulrajani et al. 2017), as well as sequential data such as handwriting and speech (Bayer and Osendorfer 2015; Chung et al. 2015). They specify a conditional distribution of observed data, given a set of hidden (latent) variables. The stochastic gradient varia-tional Bayes (SGVB) algorithm (Kingma and Welling 2014; Rezende, Mohamed, and Wierstra 2014) has made (approximate) maximum likelihood learning possible on a large scale in models where the true posterior distribution of the latent variables is not tractable.