Learning Tractable Distributions Of Language Model Continuations