A latent-observed dissimilarity measure

Models with latent variables have been proposed and investigated for explaining, understanding, or classifying observed data. If a model is a generative model, observed data are modeled to be as if they were generated by latent variables through parameterized probability distributions. Popular criteria for learning generative models include likelihood or posterior probability, which both evaluate the probability of the given observed data or parameters. Another kind of criteria is mutual information. Mutual information has been used to learn nonlinear generative models [14] in which relationships between observed and latent variables are directly evaluated. It has also been used to learn linear encoding (recognition) models [2, 12]. The relationships between observed and latent variables have greater importance in more complex generative models, e.g., deep learning models [6, 9]. In the pre-training of deep belief networks (DBNs), one of the models or techniques of deep learning, posterior samples of latent variables in the lower layer are used as samples of observed variables in the next, higher layer. For successive layer learning to be possible, latent variables should possess properties that enable such learning.