Reviews: Can Unconditional Language Models Recover Arbitrary Sentences?

This paper explores an interesting question: if we are allowed certain control over the input to a pre-trained language model, can we get it to return an arbitrary sentence? The control given is a vector z associated with the sentence, which is added as a bias to the hidden state at each timestep. In forward estimation, gradient descent is used to find the optimal z to "bias" the decoder towards a given sentence. In backward estimation, a given z is decoded to find the MAP sentence it encodes (which is intractable in general, so the authors use beam search). The authors analyze the "effective dimensionality" of a sentence space given a recoverability threshold tau; that is, what's the smallest dimension such that at most a tau-fraction of sentence fail to be encoded?