Attention for Inference Compilation

Work in progress generative models written as programs. Conditions on these random variables are imposed through observe statements, while the sample statements define latent variables we seek to draw inference about. Common to the different languages is the existence of an inference backend, which contains one or more general inference methods. Recent research has addressed the task of making repeated inference less computationally expensive, by using upfront computation to reduce the cost of later executions, an approach known as amortized inference (Gershman and Goodman, 2014). One new method called inference compilation (IC) (Le et al., 2017) enables fast inference on arbitrarily complex and non-differentiable generative models. The approximate posterior distribution it learns can be combined with importance sampling at inference time, so that inference is asymptotically correct.