Empirical Entropy Manipulation for Real-World Problems

No finite sample is sufficient to determine the density, and therefore the entropy, of a signal directly. Some assumption about either the functional form of the density or about its smoothness is necessary. By far the most common approach is to assume that the density has a parametric form. By contrast we derive a differential learning rule called EMMA that optimizes entropy by way of kernel density estimation. En(cid:173) tropy and its derivative can then be calculated by sampling from this density estimate.