On the Generative Utility of Cyclic Conditionals

We study whether and how can we model a joint distribution p(x,z) using two conditional models p(x z) and q(z x) that form a cycle. This is motivated by the observation that deep generative models, in addition to a likelihood model p(x z), often also use an inference model q(z x) for extracting representation, but they rely on a usually uninformative prior distribution p(z) to define a joint distribution, which may render problems like posterior collapse and manifold mismatch. To explore the possibility to model a joint distribution using only p(x z) and q(z x), we study their compatibility and determinacy, corresponding to the existence and uniqueness of a joint distribution whose conditional distributions coincide with them. We develop a general theory for operable equivalence criteria for compatibility, and sufficient conditions for determinacy. Based on the theory, we propose a novel generative modeling framework CyGen that only uses the two cyclic conditional models.