Interpolating in t-SNE space with Natural Neighbors

In deep learning, we often work with spaces that have hundreds or thousands of dimensions, such as the latent space of neural networks. As we have seen, visualizing these spaces requires projecting the data in 2 or 3 dimension. A very common tool for that is the t-SNE (t-distributed stochastic neighbor embedding), developped by Sam Roweis and Geoffrey Hinton. However, despite its great visualization capacities, t-SNE can be misleading in several ways. Namely, the size, density and distance of clusters doesn't necessarily convey relevant information, and should be discarded.