Disentanglement Analysis with Partial Information Decomposition

When we recognize objects, sounds, sentences, or whatever sensible, we quickly comprehend how it differs from others in properties that may individually vary across instances, such as color, shape, texture, pitch, rhythm, writing style, tone, etc. Such interpretable factors of variation are useful to understand what constitutes the variations of data and to manipulate data generation when a generative process is available. Disentanglement is a guiding principle of designing a learned representation separable into parts individually capture the underlying factors of variation. The concept is originally concerned as an inductive bias in representation learning towards obtaining representations aligned with the underlying factors of variation in data (Bengio et al., 2013) and has been applied to controlling otherwise unstructured representations of data from several domains, e.g., images (Karras et al., 2019; Esser et al., 2019), text (Hu et al., 2017), and audio (Hsu et al., 2019) to name just a few. While the concept is appealing, a concrete definition of disentanglement is not trivial. Most of the existing studies after Higgins et al. (2017) proposed generative learning methods that encourage latent variables to be marginally independent from each other; however, it is still not clear if that is the ultimate direction for better disentanglement (Higgins et al., 2018). To understand disentanglement, it is crucial to design disentanglement metrics that measure how representations disentangle the true generative factors, as it is not trivial as well to define such metrics (Higgins et al., 2017; Kim & Mnih, 2018; Chen et al., 2018; Eastwood & Williams, 2018).