Supplementary Materials Posterior Collapse and Latent Variable Non-identifiability A Examples of posterior collapse continued

We consider classical probabilistic principal component analysis ( PPCA) and show that its local latent variables can suffer from posterior collapse at maximum likelihood parameter values (i.e. 's are the latent variables of interest and others's are not (fully) identifiable in this However, it is nearly non-identifiable. While the two data generating clusters are different, they are very similar to each other because they overlap. We first define the most general form of LIDV AE . The key difference is in Eq. 19, where the classical VA E uses an arbitrary function General LIDV AE emulate many existing VA E . This general LIDV AE also subsumes the Bernoulli mixture model, which is a common variant of LIDGMV AE for the MNIST data. Moreover, for any data distribution generated by the classical VA E ( Eqs. 17 to 19), there exists an LIDV AE that can generate the same distribution.