We consider classical probabilistic principal component analysis ( PPCA) and show that its local latent variables can suffer from posterior collapse at maximum likelihood parameter values (i.e. 's are the latent variables of interest and others's are not (fully) identifiable in this However, it is nearly non-identifiable. While the two data generating clusters are different, they are very similar to each other because they overlap. We first define the most general form of LIDV AE . The key difference is in Eq. 19, where the classical VA E uses an arbitrary function General LIDV AE emulate many existing VA E . This general LIDV AE also subsumes the Bernoulli mixture model, which is a common variant of LIDGMV AE for the MNIST data. Moreover, for any data distribution generated by the classical VA E ( Eqs. 17 to 19), there exists an LIDV AE that can generate the same distribution.

Inference in discrete graphical models with variational methods is difficult because of the inability to re-parameterize gradients of the Evidence Lower Bound (ELBO). Many sampling-based methods have been proposed for estimating these gradients, but they suffer from high bias or variance. In this paper, we propose a new approach that leverages the tractability of probabilistic circuit models, such as Sum Product Networks (SPN), to compute ELBO gradients exactly (without sampling) for a certain class of densities. In particular, we show that selective-SPNs are suitable as an expressive variational distribution, and prove that when the log-density of the target model is a polynomial the corresponding ELBO can be computed analytically. To scale to graphical models with thousands of variables, we develop an efficient and effective construction of selective-SPNs with size O(kn), where n is the number of variables and k is an adjustable hyperparameter. We demonstrate our approach on three types of graphical models - Ising models, Latent Dirichlet Allocation, and factor graphs from the UAI Inference Competition. Selective-SPNs give a better lower bound than mean-field and structured mean-field, and is competitive with approximations that do not provide a lower bound, such as Loopy Belief Propagation and Tree-Reweighted Belief Propagation. Our results show that probabilistic circuits are promising tools for variational inference in discrete graphical models as they combine tractability and expressivity.

The core challenge in cooperative communication is the problem of common ground: having enough shared knowledge and understanding to successfully communicate.