A Compositional Kernel Model for Feature Learning

Deep learning has achieved remarkable success across domains such as vision, language, and science. A widely believed explanation for this success is representation learning -- also called feature learning -- the empirically observed ability of deep models to automatically extract task-relevant features from raw data, without manual engineering, to support downstream prediction [1]. This ability is generally attributed to two fundamental ingredients of deep models: (i) their compositional architecture and (ii) the use of optimization. The compositionality of the architecture endows the model with the ability to form intermediate representations of the data via composition of simple transformations. These representations are not manually defined but are learned from data by optimizing a loss function designed to minimize prediction error. However, despite the empirical success of this paradigm, our theoretical understanding of how and why such representations emerge remains fundamentally limited. In particular, it remains unclear how the interplay between compositional structure and optimization gives rise to task-aligned features -- and under what conditions this mechanism succeeds or fails. To address this gap, we study a stylized compositional model that preserves these two core ingredients of feature learning -- while remaining simple enough to enable analysis of how features are learnt during training.