Beyond the noise: intrinsic dimension estimation with optimal neighbourhood identification