Null/No Information Rate (NIR): a statistical test to assess if a classification accuracy is significant for a given problem

In many research contexts, especially in the biomedical field, after studying and developing a classification system a natural question arises: "Is this accuracy enough high?", or better, "Can we say, with a statistically significant confidence, that our classification system is able to solve the problem"? To answer to this question we can use the statistical test described in this paper, which is referred in some cases as NIR (No Information Rate or Null Information Rate). In many research contexts, especially in the biomedical field, we have a classification problem for which we develop a classification system. Then we evaluate the performances of such system by measuring its classification accuracy (or error), typically estimated with a Cross Validation protocol. In particular we have a dataset, which contains objects for which we know the true category, and we split the dataset in two separated sets: one, called training set, is used to build the classifier and the other, called testing set, is used to test it: we classify the objects in the testing set with the trained classifier and we count the number of times our classifier provides a correct answer, i.e. the answer of the classifier on a given object is identical to its true label.