Suppose you are working on a machine learning project, for which you want to predict if a set of patients have or not a mortal disease, based on several features on your dataset as blood pressure, heart rate, pulse and others. Sounds like a serious project, for which you'll need to really trust your model and predictions, right? That's why you got hundreds of samples, that your local hospital very gently allowed you to collect, given the importance and the seriousness of the topic. But how do you know if your sample is representative of the whole population? And how can we know how much difference might be reasonable?

In an earlier post, I described work that I had initially done as an Insight Data Engineering Fellow. That work, now merged into Flink's master branch, was to do an efficient exact k-nearest neighbors (KNN) query using quadtrees. I have since worked on an approximate version of the KNN algorithm, and I will discuss one method I used for the approximate version using Z-value based hashing. For large and high dimensional data sets, an exact k-nearest neighbors query can become infeasible. There are many algorithms that reduce the dimensionality of the points by hashing them to lower dimensions.