Paraphrase generation is a longstanding important problem in natural language processing. Recent progress in deep generative models has shown promising results on discrete latent variables for text generation. Inspired by variational autoencoders with discrete latent structures, in this work, we propose a latent bag of words (BOW) model for paraphrase generation. We ground the semantics of a discrete latent variable by the target BOW. We use this latent variable to build a fully differentiable content planning and surface realization pipeline. Specifically, we use source words to predict their neighbors and model the target BOW with a mixture of softmax.

Thus paper presents a model where a latent bag-of-words inform a paraphrase generation model. For each source words, the authors compute a multinomial over "neighbor" vocabulary words; this then yields a bag-of-words by a mixture of softmaxes over these neighbors. In the generative process, a set of words is drawn from this distribution, then their word embeddings are averaged to form input to the decoder. During training, the authors use a continuous relaxation of this with Gumbel top-k sampling (a differentiable way to sample k of these words without replacement). The words are averaged and fed into the LSTM's initial state.

The paper proposes a two-stage model for sentence-level paraphrase generation, trained end-to-end. The first stage is content planning (specifically predicting a'latent' bag of keywords). The second one is the surface realization stage (forming a sentence relying on the keywords). The model is interesting and novel. The evaluation is sufficiently convincing (the author response, I believe, addressed initial concerns of the reviewer 1).

Paraphrase generation is a longstanding important problem in natural language processing. Recent progress in deep generative models has shown promising results on discrete latent variables for text generation. Inspired by variational autoencoders with discrete latent structures, in this work, we propose a latent bag of words (BOW) model for paraphrase generation. We ground the semantics of a discrete latent variable by the target BOW. We use this latent variable to build a fully differentiable content planning and surface realization pipeline.

Paraphrase generation is a longstanding important problem in natural language processing. Recent progress in deep generative models has shown promising results on discrete latent variables for text generation. Inspired by variational autoencoders with discrete latent structures, in this work, we propose a latent bag of words (BOW) model for paraphrase generation. We ground the semantics of a discrete latent variable by the target BOW. We use this latent variable to build a fully differentiable content planning and surface realization pipeline.