What's the Difference Between a Metric and a Loss Function?

Have you been using your loss function for evaluating your machine learning system's performance? That's a mistake, but don't worry, you're not alone. It's a widespread misunderstanding that may have something to do with software defaults, college course format, and decision-maker absenteeism in AI. In this article, I'll explain why you need two separate model scoring functions for evaluation and optimization… and possibly a third one for statistical testing. Throughout data science, you'll see scoring functions (like the MSE, for example) being used for three main purposes: These three are subtly -- but importantly -- different from one another, so let's take a deeper look at what makes a function "good" for each purpose.