Mixture of Discrete Normalizing Flows for Variational Inference

This has made it easier to step outside the rigid model and approximation families designed based on computational tractability, and to switch to flexible distributions parameterized by neural networks, including normalizing flows relying on invertible transformation of simple base distributions [21]. Models with discrete latent variables, however, remain problematic due to non-differentiability of the sampling operation that prevents efficient optimization of expectations over the approximation. Hence in practice, we still largely resort to model-specific algorithms that are tedious to extend already for minor variants of the model, analytic marginalization of the discrete latent variables (e.g., mixture models and LDA in Stan [7]), continuous relaxations like the concrete distribution [19, 13], or (semi-)implicit approximations that do not support probability evaluation [34, 30]. Normalizing flows [25, 15, 21], suitable for learning flexible posterior approximations for continuous variables, have recently been generalized also for discrete categorical [31] and ordinal [12] variables. However, the discrete variants have only been applied in generative modeling of discrete observations. Even though discrete normalizing flows for categorical distributions (DNF) retain the property of differentiable Monte Carlo estimates, they are not very suitable for variational approximation due to their limited expressive power. As we will show later, a DNF can only move probability mass around and hence relies extremely strongly on use of base distributions that are already expressive (and of the same dimension as the final distribution). For generative modeling this can be satisfied, for example by using recurrent neural networks as base distributions in the case of language modeling [31], but for modeling latent variables there are no easy ways of learning strong base distributions. We improve the expressive power of DNFs by constructing mixtures of categorical discrete normalizing flows (MDNF).