Going Deeper with Spectral Decompositions

Eigen and singular decompositions are ubiquitous in applied mathematics. They can serve as a basis to define good features in machine learning pipelines (Belkin and Niyogi, 2003; Coifman and Lafon, 2006; Balestriero and LeCun, 2022), while a set of good features naturally define pullback distances on the original data. Those features and distances are naturally referred to as "spectral embeddings" and "spectral distances". The latter are thought to provide meaningful geometries on the data, which explain their uses for clustering (Belkin and Niyogi, 2004; Schubert et al., 2018), as well as diffusion models (Chen and Lipman, 2023). In the machine learning community, spectral decompositions are usually derived from the eigen decompositions of different graph Laplacians built on top of the data (Chung, 1997; Zhu et al., 2003; Ham et al., 2004). However, those methods are known to scale poorly with the input dimension (Bengio et al., 2006; Singer, 2006; Hein et al., 2007), although they had applications in many different fields, such as molecular simulation (Glielmo et al., 2021), acoustics (Bianco et al., 2019) or the study of gene interaction (van Dijk et al., 2018). In this paper, we suggest a different approach to approximate the spectral decompositions of a large class of operators. Our method consists in restricting the study of infinite-dimensional operators on a basis of simple functions, which is usually referred to as Galerkin, Ritz or Raleigh methods (Singer, 1962), if not Bubnov or Petrov (Fluid Dynamics, 2012), depending on the research community. We make the following contributions.