UC Berkeley Uses a Causal Perspective to Formalise the Desiderata for Representation Learning

Representation learning is used to summarize essential features of high-dimensional data and turn them into lower-dimensional representations with desirable properties. A popular method for this is the heuristic approach, which fits a neural network that maps from the high dimensional data to a set of labels, taking the top layer of the neural network as the representation of the inputs. However, such heuristic approaches often end up capturing spurious features that do not transfer well; or finding entangled dimensions that are uninterpretable. And while non-spuriousness or disentanglement are natural desiderata of representations, they are difficult to evaluate and optimize over algorithmically. To address this issue, a new study by UC Berkeley researchers Yixian Wang and Michael I. Jordon takes a causal perspective on representation learning, which enables the formalization of non-spuriousness, efficiency and disentanglement representation learning desiderata using causal notions.