Recursive KL Divergence Optimization: A Dynamic Framework for Representation Learning

Contrastive methods, dimensionality reduction algorithms like t-SNE, and clustering objectives such as k-Means all implicitly or explicitly define distributions over neighborhoods and minimize some divergence between them. The Information Contrastive Learning (I-Con) framework recently unified many such approaches by expressing them as the minimization of KL divergence between a fixed supervisory distribution p (j | i) and a learned distribution q ( j |i) over data neighborhoods [1]. However, I-Con treats this KL alignment statically, as if each point-wise loss were independent. In this paper, we propose a deeper view: that representation learning is fundamentally a process of recursive divergence minimization across a structured field of conditional distributions. Each neighborhood distribution depends on prior learned representations, forming a dynamic system that we call Recursive KL Divergence Optimization (RKDO). While the exponential moving average (EMA) recursion we employ has been used in several well-known self-supervised and semi-supervised methods such as Temporal Ensembling [2], Mean Teacher [3], and momentum-based frameworks like MoCo [4], BYOL [5], and DINO [6], our novel contribution lies in applying this recursive structure to the entire response field (the joint conditional distribution over representation pairs), rather than to individual weights or per-sample predictions. RKDO captures the temporal dynamics of representation learning that are absent in static frameworks, with significant implications for optimization efficiency. Our contributions include: A new theoretical framework that generalizes representation learning as recursive alignment of conditional distributions across the entire response field Mathematical formulations showing how RKDO captures temporal dynamics absent in static frameworks, with a formal proof of linear-rate convergence under this recursion Empirical evidence that RKDO's recursive approach results in significantly lower loss values (approximately 30% reduction across all tested datasets) Demonstration that RKDO requires 60-80% fewer computational resources (training epochs) to achieve results comparable to longer I-Con training Analysis of the trade-offs between optimization efficiency and generalization in recursive versus static approaches Our experiments suggest that while I-Con effectively represents a unified view of many typical representation learning approaches, RKDO can provide substantial efficiency improvements: achieving comparable optimization objectives with approximately 30% lower loss values, while potentially reducing computational requirements by 60-80% in the specific scenarios we studied. 2 Background and Related Work The KL divergence [7] is a foundational object in representation learning.