Variational Message Passing with Structured Inference Networks

Recent efforts on combining deep models with probabilistic graphical models are promising in providing flexible models that are also easy to interpret. We propose a variational message-passing algorithm for variational inference in such models. First, we propose structured inference networks that incorporate the structure of the graphical model in the inference network of variational auto-encoders (V AE). Second, we establish conditions under which such inference networks enable fast amortized inference similar to V AE. Finally, we derive a variational message passing algorithm to perform efficient natural-gradient inference while retaining the efficiency of the amortized inference. By simultaneously enabling structured, amortized, and natural-gradient inference for deep structured models, our method simplifies and generalizes existing methods. To analyze real-world data, machine learning relies on models that can extract useful patterns. Deep Neural Networks (DNNs) are a popular choice for this purpose because they can learn flexible representations. Another popular choice are probabilistic graphical models (PGMs) which can find interpretable structures in the data. Recent work on combining these two types of models hopes to exploit their complimentary strengths and provide powerful models that are also easy to interpret (Johnson et al., 2016; Krishnan et al., 2015; Archer et al., 2015; Fraccaro et al., 2016). To apply such hybrid models to real-world problems, we need efficient algorithms that can extract useful structure from the data. For deep learning, stochastic-gradient methods are the most popular choice, e.g., those based on back-propagation.