Machine Learning: An In-Depth Guide – Model Performance and Error Analysis

Welcome to the fourth article in a five-part series about machine learning. In this article, we will take a deeper dive into model evaluation and performance metrics, and potential prediction-related errors that one may encounter. Before digging deeper into model performance and error types, we must first discuss the concept of residuals and errors for regression, positive and negative classifications for classification problems, and in-sample versus out-of-sample measurements. Any reference to models, metrics, or errors computed with respect to the data used to train, validate, or tune a predictive model (i.e., data you have) is called in-sample. Conversely, reference to test data metrics and errors, or new data in general is called out-of-sample (i.e., data you don't have). Recall that regression involves predicting a continuous valued output (response) based on some set of input variables (features/predictors).