A Similarity Measure Between Functions with Applications to Statistical Learning and Optimization

Quantifying the closeness between two functions is an essential part of many studies in statistical learning and optimization. For example, in empirical risk minimization, the convergence rate of an empirical minimizer is often derived from studying the concentration of the empirical risk around its population version (Bousquet et al., 2004; Bartlett et al., 2005; Wainwright, 2019). In non-stationary online optimization, the discrepancy between loss functions in different periods reflects the variation of the underlying environment (Besbes et al., 2015; Jadbabaie et al., 2015; Chen et al., 2019). In this note, we present a novel measure of similarity between functions that unifies several existing notions of functional similarity, and illustrate its use in statistical learning and optimization. The measure was first proposed by Huang and Wang (2023) for studying the problem of online statistical learning under non-stationarity.