Statistical Tests for Comparing Classification Algorithms

Comparing prediction methods to define which one should be used for the task at hand is a daily activity for most data scientists. Usually, one will have a pool of classification models and will validate them using cross-validation to define which one is best. Another goal, however, is not to compare classifiers, but the learning algorithms themselves. The idea is: given this task (data), which learning algorithm (KNN, SVM, Random Forests, etc) will generate more accurate classifiers on a dataset of size D? As we will see, every method presented here has some pros and cons. However, the first intuition of using a two proportions test can lead to some really bad results.