Variational Information Bottleneck on Vector Quantized Autoencoders

In this paper, we provide an information-theoretic interpretation of the Vector Quantized-Variational Autoencoder(VQ-VAE). We show that the loss function of the original VQ-VAE [1] can be derived from the variational deterministic information bottleneck (VDIB) principle [2]. On the other hand, the VQ-VAE trained by the Expectation Maximization (EM) algorithm [3] can be viewed as an approximation to the variational information bottleneck(VIB) principle [4]. I Introduction The recent advances of variational autoencoder(VAE) provide new unsupervised approaches to learn hidden structure of the data [5]. The variational autoencoder is a powerful generative model which allows inference of the learned latent representation. However, the classic VAEs are prone to the "posterior collapse "phenomenon that the latent representations are ignored due to the powerful decoder. Vector quantized variational autoencoder (VQ-VAE) learns discrete representations by incorporating the idea of vector quantization into the bottleneck stage and the "posterior collapse "can be avoided [1].