Hypothesis Test for Comparing Machine Learning Algorithms - AnalyticsWeek

Machine learning models are chosen based on their mean performance, often calculated using k-fold cross-validation. The algorithm with the best mean performance is expected to be better than those algorithms with worse mean performance. But what if the difference in the mean performance is caused by a statistical fluke? The solution is to use a statistical hypothesis test to evaluate whether the difference in the mean performance between any two algorithms is real or not. In this tutorial, you will discover how to use statistical hypothesis tests for comparing machine learning algorithms.