Variational autoencoders (VAE) are a powerful and widely-used class of models to learn complex data distributions in an unsupervised fashion. One important limitation of VAEs is the prior assumption that latent sample representations are independent and identically distributed. However, for many important datasets, such as time-series of images, this assumption is too strong: accounting for covariances between samples, such as those in time, can yield to a more appropriate model specification and improve performance in downstream tasks. In this work, we introduce a new model, the Gaussian Process (GP) Prior Variational Autoencoder (GPPVAE), to specifically address this issue. The GPPVAE aims to combine the power of VAEs with the ability to model correlations afforded by GP priors. To achieve efficient inference in this new class of models, we leverage structure in the covariance matrix, and introduce a new stochastic backpropagation strategy that allows for computing stochastic gradients in a distributed and low-memory fashion. We show that our method outperforms conditional VAEs (CVAEs) and an adaptation of standard VAEs in two image data applications.

In experiments, VAEs trained with the new loss function generatedrealistic,high-qualityimagesamples.

Made: Maskedautoencoder fordistributionestimation. InInternational Conferenceon Machine Learning, pages 881-889, 2015.

For example, while prior work has suggested that theglobally optimal VAEsolution canlearn thecorrect manifold dimension, anecessary (butnotsufficient)condition forproducing samplesfrom the true data distribution, this has never been rigorously proven. Moreover, it remains unclear how such considerations would change when various types of conditioning variablesare introduced, or when the data support is extended to a union of manifolds (e.g., as is likely the case for MNIST digits and related). In this work, we address these points by first proving that VAE global minima are indeed capable of recovering the correct manifold dimension.

Wehere define the appearance as being the factors of data that are left after transformingxby its perspective.

Inference networks oftraditional Variational Autoencoders (VAEs) aretypically amortized, resulting in relatively inaccurate posterior approximation compared to instance-wise variational optimization. Recent semi-amortized approaches were proposedtoaddress thisdrawback; however,theiriterativegradient update procedures can be computationally demanding.

In this section, we review some key results on the Wasserstein distance. Wpp Rπ(t,θi),Rρ(t,θi), (4) where the approximation comes from using Monte-Carlo integration by samplingθi uniformly in SD 1 [2]. M,M is the number of points used to approximate the integral. Calculating the Wasserstein distance with the empirical distribution function is computationally attractive. To do that, we first sortxms in an ascending order, such thatxi[m] xi[m+1], where i[m]istheindexofthesortedxms. Hamiltonian Monte Carlo (HMC)[24]isahighly-efficient MarkovChain Monte Carlo (MCMC) method used to generate samples from the posteriorw p(w|y).

On that basis, we hypothesize that these two problems stem from the conflict between the KL regularization inELBo andthefunction definition oftheprior distribution. Assuch, wepropose a novel regularization to substitute the KL regularization in ELBo for VAEs, which isbased on the density gapbetween the aggregated posterior distribution and the prior distribution.

Irina Higgins, Shakir Mohamed, constrained International 2017.