Statistics for Evaluating Machine Learning Models

The skill or prediction error of a model must be estimated, and as an estimate, it will contain error. This is made clear by distinguishing between the true error of a model and the estimated or sample error. One is the error rate of the hypothesis over the sample of data that is available. The other is the error rate of the hypothesis over the entire unknown distribution D of examples.