When Getting It Right Gets It Wrong

In a previous post I briefly touched on the problem with overfitting, which is loosely defined as a machine learning model that memorizes a training data set and thus provides high accuracy for predictions using it, but then performs poorly when presented with new data -- a phenomenon known as variance. The post discussed the Random Forest approach using bootstrap aggregation to address this issue, but it begged the question: "Why does intentionally producing lower-quality data sets and averaging across their results produce better predictions?" Reality, it turns out, is messy, so intentionally introducing inaccuracy in the process of producing predictions (that's some impressive alliteration, don't you think?) usually makes them better. It's a process known as regularization. It turns out that all kinds of machine learning algorithms have overfitting risks, and they way you regularize depends on the model you're trying to fit.