PRI-VAE: Principle-of-Relevant-Information Variational Autoencoders

Although substantial efforts have been made to learn disentangled representations under the variational autoencoder (VAE) framework, the fundamental properties to the dynamics of learning of most VAE models still remain unknown and under-investigated. We then present an information-theoretic perspective to analyze existing VAE models by inspecting the evolution of some critical information-theoretic quantities across training epochs. Our observations unveil some fundamental properties associated with VAEs. Empirical results also demonstrate the effectiveness of PRI-VAE on four benchmark data sets. A central goal for representation learning models is that the resulting latent representation should be compact yet disentangled. Compact requires the representation z does not contain any nuance factors in the input signal x that are not relevant for the desired response y [1], whereas disentangled means that z is factorizable and has consistent semantics associated to different generating factors of the underlying data generation process. Yanjun Li, Jose C. Principe and Dapeng Wu are with the NSF Center for Big Learning, University of Florida, U.S.A (email: yanjun.li@ufl.edu, Shujian Yu is with the Machine Learning Group, NEC Laboratories Europe, Germany (email: Shujian.Yu@neclab.eu). To whom correspondence should be addressed.