Optimal Manifold Representation of Data: An Information Theoretic Approach

We introduce an information theoretic method for nonparametric, non- linear dimensionality reduction, based on the infinite cluster limit of rate distortion theory. By constraining the information available to manifold coordinates, a natural probabilistic map emerges that assigns original data to corresponding points on a lower dimensional manifold. With only the information-distortion trade off as a parameter, our method de- termines the shape of the manifold, its dimensionality, the probabilistic map and the prior that provide optimal description of the data. Some data sets may not be as complicated as they appear. Consider the set of points on a plane in Figure 1.