Stochastic Thermodynamics of Learning Parametric Probabilistic Models

Starting from nearly half a century ago, physicists began to learn that information is a physical entity [1, 2, 3]. Today, the information-theoretic perspective has significantly impacted various fields of physics, including quantum computing [4], cosmology [5], and thermodynamics [6]. Simultaneously, recent years have witnessed the remarkable success of an algorithmic approach known as machine learning, which is adept at learning information from data. This paper is propelled by a straightforward proposition: if "information is physical", then the process of learning information must inherently be a physical process. The concepts of memory, prediction, and information exchange between subsystems have undergone extensive exploration within the realms of Thermodynamics of Information [6] and Stochastic Thermodynamics [7]. For instance, Still et al. [8] delved into the thermodynamics of prediction. And, the role of information exchange between thermodynamic subsystems has been studied by Sagawa and Ueda [9], and Esposito et al. [10]. This rich toolbox of thermodynamic of information is our main venue to study physics of machine learning process, with motivation to assess the information content of the learning process. The type of machine learning problems we consider in this study encompasses any algorithmic approach that evolves a Parametric Probabilistic Model (PPM), or simply the model, towards a desirable target distribution through gradientbased loss function minimization.