Local intrinsic dimensionality estimators based on concentration of measure
Bac, Jonathan, Zinovyev, Andrei
--Intrinsic dimensionality (ID) is one of the most fundamental characteristics of multidimensional data point clouds. Knowing ID is crucial to choose the appropriate machine learning approach as well as to understand its behavior and validate it. ID can be computed globally for the whole data distribution, or estimated locally in a point. In this paper, we introduce new local estimators of ID based on linear separability of multidimensional data point clouds, which is one of the manifestations of concentration of measure. We empirically study the properties of these measures and compare them with other recently introduced ID estimators exploiting various effects of measure concentration. Observed differences in the behaviour of different estimators can be used to anticipate their behaviour in practical applications. Datasets used in applications of machine learning frequently contain objects characterized by thousands and even millions of features.
Jan-31-2020