Review for NeurIPS paper: Analytical Probability Distributions and Exact Expectation-Maximization for Deep Generative Networks

Summary and Contributions: Deep generative models (DGMs), specifically variational autoencoders (VAEs), currently rely on variational inference and stochastic optimization of a lower bound to maximize likelihood since the analytic likelihood cannot be computed in general. This paper shows that in fact the likelihood can be computed analytically and maximized with analytic expectation maximization (EM) updates when the network uses affine piecewise nonlinearities like ReLU and leaky-ReLU. The key insight is that these networks induces a partition of the latent space that can be handled tractably when the prior and likelihood are both Gaussian. This paper analytically derives the posterior distribution, the marginal distribution, the expectation of the complete likelihood (for the E step), and the updates to the parameters (for the M step). These novel derivations allows the authors to perform EM on DGMs for the first time.