The preoccupation with test error in applied machine learning

"Predictive accuracy on test sets is the criterion for how good the model is." The quote above may be one of the most important observations, from one of the most important papers, in data science. So forgive me because I am not worthy, but I propose a reinterpretation of this philosophy for the commercial practice of applied machine learning in 2016. The technology exists now, be it purchased or built in-house, to directly measure the monetary value that a machine model is generating. This monetary value should be the criterion for selecting and deploying a commercial machine learning model, not its performance on old, static test data sets. In the worst cases, I've seen organizations choose models purely based on hype, or the shiny appeal of novelty (often buttressed by a blog post or whitepaper with impressive test data performances).