Handling the (ground) truth: Control group based KPIs

In my previous post, I introduced a framework to select the "right" KPIs to evaluate your machine learning models. When dealing with a classification problem, the chosen KPI will usually measure the rate of mistakes over time, as a point estimate (e.g. In this post, I will touch on one of the main challenges in many real world problems: how to know whether your model is actually correct or not. In supervised learning models, getting feedback on the model's decisions is crucial for both model training and evaluation. But getting such feedback can sometimes be very challenging or require a lot of resources.