Influence Functions for Machine Learning: Nonparametric Estimators for Entropies, Divergences and Mutual Informations

Entropies, divergences, and mutual informations are classical information-theoretic quantities that play fundamental roles in statistics, machine learning, and across the mathematical sciences. In addition to their use as analytical tools, they arise in a variety of applications including hypothesis testing, parameter estimation, feature selection, and optimal experimental design. In many of these applications, it is important to estimate these functionals from data so that they can be used in downstream algorithmic or scientific tasks. In this paper, we develop a recipe for estimating statistical functionals of one or more nonparametric distributions based on the notion of influence functions. Entropy estimators are used in applications ranging from independent components analysis [Learned-Miller and John, 2003], intrinsic dimension estimation [Carter et al., 2010] and several signal processing applications [Hero et al., 2002].