Neural network-based generative language models like ELMo and BERT can work effectively as general purpose sentence encoders in text classification without further fine-tuning. Is it possible to adapt them in a similar way for use as general-purpose decoders? For this to be possible, it would need to be the case that for any target sentence of interest, there is some continuous representation that can be passed to the language model to cause it to reproduce that sentence. We set aside the difficult problem of designing an encoder that can produce such representations and, instead, ask directly whether such representations exist at all. To do this, we introduce a pair of effective, complementary methods for feeding representations into pretrained unconditional language models and a corresponding set of methods to map sentences into and out of this representation space, the reparametrized sentence space. We then investigate the conditions under which a language model can be made to generate a sentence through the identification of a point in such a space and find that it is possible to recover arbitrary sentences nearly perfectly with language models and representations of moderate size.

This paper explores an interesting question: if we are allowed certain control over the input to a pre-trained language model, can we get it to return an arbitrary sentence? The control given is a vector z associated with the sentence, which is added as a bias to the hidden state at each timestep. In forward estimation, gradient descent is used to find the optimal z to "bias" the decoder towards a given sentence. In backward estimation, a given z is decoded to find the MAP sentence it encodes (which is intractable in general, so the authors use beam search). The authors analyze the "effective dimensionality" of a sentence space given a recoverability threshold tau; that is, what's the smallest dimension such that at most a tau-fraction of sentence fail to be encoded?

While we now understand how we can pretrain text encoders or non-conditional language models, the important open question is figuring out a method for pretraining (or using pretrained) decoders in seq2seq models. While several proposals have been made, neither was particularly successful. This paper does not deliver this either but it answers a very natural question anyone working on this problem would like to ask -- can we even steer a pretrained language models so that it generates a given sequence? I (as well as) reviewers found the paper (very) interesting: the study is well executed, well written and provides new insights into the properties and limitations of pretrained language models. There is consensus that the paper should be accepted.

Neural network-based generative language models like ELMo and BERT can work effectively as general purpose sentence encoders in text classification without further fine-tuning. Is it possible to adapt them in a similar way for use as general-purpose decoders? For this to be possible, it would need to be the case that for any target sentence of interest, there is some continuous representation that can be passed to the language model to cause it to reproduce that sentence. We set aside the difficult problem of designing an encoder that can produce such representations and, instead, ask directly whether such representations exist at all. To do this, we introduce a pair of effective, complementary methods for feeding representations into pretrained unconditional language models and a corresponding set of methods to map sentences into and out of this representation space, the reparametrized sentence space.

Neural network-based generative language models like ELMo and BERT can work effectively as general purpose sentence encoders in text classification without further fine-tuning. Is it possible to adapt them in a similar way for use as general-purpose decoders? For this to be possible, it would need to be the case that for any target sentence of interest, there is some continuous representation that can be passed to the language model to cause it to reproduce that sentence. We set aside the difficult problem of designing an encoder that can produce such representations and, instead, ask directly whether such representations exist at all. To do this, we introduce a pair of effective, complementary methods for feeding representations into pretrained unconditional language models and a corresponding set of methods to map sentences into and out of this representation space, the reparametrized sentence space.