GAIA: Categorical Foundations of Generative AI

Figure 1: We propose a hierarchical Generative AI Architecture (GAIA) using higher-order category theory. Generative AI has become a dominant paradigm for building intelligent systems in the last few years, ranging from large language models developed with the widely used Transformer model Vaswani et al. (2017), or more recently with the structured state space sequence models Gu et al. (2022); Yin et al. (2023), and with the growing use of image diffusion algorithms Song and Ermon (2019); Yin et al. (2023). We can broadly define the problem of generative AI as the construction, maintenance, and deployment of foundation models Bommasani et al. (2022), a storehouse of human knowledge that provides the basic infrastructure for AI across some set of applications. A fundamental question, therefore, to investigate is to study the mathematical basis for foundation models. We propose a mathematical framework for a Generative AI Architecture (GAIA) (see Figure 1) based on the hypothesis that category theory MacLane (1971); Riehl (2017); Lurie (2009) provides a universal mathematical language for foundation models.