Generation through the lens of learning theory

Over the past 50 years, predictive machine learning has been a cornerstone for both theorists and practitioners. Predictive tasks like classification and regression have been extensively studied, in both theory and practice, due to their applications to face recognition, autonomous vehicles, fraud detection, recommendation systems, etc. Recently, however, a new paradigm of machine learning has emerged: generation. Unlike predictive models, which focus on making accurate predictions of the true label given examples, generative models aim to create new examples based on observed data. For example, in language modeling, the goal might be to generate coherent text in response to a prompt, while in drug development, one might want to generate candidate molecules. In fact, generative models have already been applied to these tasks and others [Zhao et al., 2023, Jumper et al., 2021]. The vast potential of generative machine learning has spurred a surge of research across diverse fields like natural language processing [Wolf et al., 2020], computer vision [Khan et al., 2022], and computational chemistry/biology [Vanhaelen et al., 2020]. Despite this widespread adoption, the theoretical foundations of generative machine learning lags far behind its predictive counterpart. While prediction has been extensively studied by learning theorists through frameworks like PAC and online learning [Shalev-Shwartz and Ben-David, 2014, Mohri et al., 2012, Cesa-Bianchi and Lugosi, 2006], generative machine learning has, for the most, part