Optimal Manifold Representation of Data: An Information Theoretic Approach

We introduce an information theoretic method for nonparametric, nonlinear dimensionalityreduction, based on the infinite cluster limit of rate distortion theory. By constraining the information available to manifold coordinates, a natural probabilistic map emerges that assigns original data to corresponding points on a lower dimensional manifold. With only the information-distortion trade off as a parameter, our method determines theshape of the manifold, its dimensionality, the probabilistic map and the prior that provide optimal description of the data.