Understanding Influence Functions and Datamodels via Harmonic Analysis

It is often of great interest to quantify how the presence or absence of a particular training data point affects the trained model's performance on test data points. Influence functions is a classical idea for this [Jaeckel, 1972, Hampel, 1974, Cook, 1977] that has recently been adapted to modern deep models and large datasets Koh and Liang [2017]. Influence functions have been applied to explain predictions and produce confidence intervals [Schulam and Saria, 2019], investigate model bias [Brunet et al., 2019, Wang et al., 2019], estimate Shapley values [Jia et al., 2019, Ghorbani and Zou, 2019], improve human trust [Zhou et al., 2019], and craft data poisoning attacks [Koh et al., 2019]. Influence actually has different formalizations. The classic calculus-based estimate (henceforth referred to as continuous influence) involves conceptualizing training loss as a weighted sum over training datapoints, where the weighting of a particular datapoint z can be varied infinitesimally.