Interpretable Kernel Representation Learning at Scale: A Unified Framework Utilizing Nyström Approximation

Kernel methods provide a theoretically grounded framework for non-linear and non-parametric learning, with strong analytic foundations and statistical guarantees. Y et, their scalability has long been limited by prohibitive time and memory costs. While progress has been made in scaling kernel regression, no framework exists for scalable kernel-based representation learning, restricting their use in the era of foundation models where representations are learned from massive unlabeled data. We introduce KREPES--a unified, scalable framework for kernel-based representation learning via Nystr om approximation. KREPES accommodates a wide range of unsupervised and self-supervised losses, and experiments on large image and tabular datasets demonstrate its efficiency. Crucially, KREPES enables principled interpretability of the learned representations, an immediate benefit over deep models, which we substantiate through dedicated analysis. Representation learning aims to uncover meaningful low-dimensional representations that generalize across tasks by capturing underlying data structure in a compact, expressive form. Earlier approaches relied on handcrafted features, e.g., SIFT, SURF (Park et al., 2022) for images or bag-of-words for text, which were domain-specific and required expert knowledge. In contrast, learned representations optimized directly from data have become dominant for their adaptability.