Evaluating Classification Models, Part 3

This series differs from other discussions of evaluation metrics for classification models in that it aims to provide a systematic perspective. Rather than providing a laundry list of individual metrics, it situates those metrics within a fairly comprehensive family and explains how you can choose a member of that family that is appropriate for your use case. This post explains how the three weighted "Pythagorean means" (arithmetic, geometric, and harmonic) of precision and recall encode preferences over models. Suppose we build two different models, and one has better precision while the other has better recall. To choose between these models, we need to decide whether the gain from 90.8% precision to 91.5% precision that we get by going from Model A to Model B is enough to offset a loss from 99% recall to 97% recall.