Does a Variational AutoEncoder (VAE) consistently encode typical samples generated from its decoder? This paper shows that the perhaps surprising answer to this question is `No'; a (nominally trained) VAE does not necessarily amortize inference for typical samples that it is capable of generating. We study the implications of this behaviour on the learned representations and also the consequences of fixing it by introducing a notion of self consistency. Our approach hinges on an alternative construction of the variational approximation distribution to the true posterior of an extended VAE model with a Markov chain alternating between the encoder and the decoder. The method can be used to train a VAE model from scratch or given an already trained VAE, it can be run as a post processing step in an entirely self supervised way without access to the original training data. Our experimental analysis reveals that encoders trained with our self-consistency approach lead to representations that are robust (insensitive) to perturbations in the input introduced by adversarial attacks. We provide experimental results on the ColorMnist and CelebA benchmark datasets that quantify the properties of the learned representations and compare the approach with a baseline that is specifically trained for the desired property.

Additional Feedback: 1) Figure 1 does not aid your case that this phenomenon exists. While I am familiar with this line of research from other work and thus know that VAEs and other encoders drift, a better argument should be made to the reader, both in the introduction and in Figure 1. 2) The result that the AVAE condition makes representations more robust is empirical. This isn't a problem (empirical results are good too), but it seems almost independent to the theoretical frame and intuition of the work. I understand there are space constraints (this is a very full paper), but an analysis of why VAE vulnerabilities to adversarial attacks are mitigated by the AVAE condition would be helpful. Notably, \varepsilon perturbation type attacks are obviously not constrained to any data manifold (or, in a probabilistic sense, may move data x to rare events x \varepsilon).

All reviewers agree that this is a good contribution to the VAE literature. My recommendation is to accept. Please take the reviewers' comments into account in preparing the final version of the paper.

Does a Variational AutoEncoder (VAE) consistently encode typical samples generated from its decoder? This paper shows that the perhaps surprising answer to this question is No'; a (nominally trained) VAE does not necessarily amortize inference for typical samples that it is capable of generating. We study the implications of this behaviour on the learned representations and also the consequences of fixing it by introducing a notion of self consistency. Our approach hinges on an alternative construction of the variational approximation distribution to the true posterior of an extended VAE model with a Markov chain alternating between the encoder and the decoder. The method can be used to train a VAE model from scratch or given an already trained VAE, it can be run as a post processing step in an entirely self supervised way without access to the original training data.