Data drifts pose a critical challenge in the lifecycle of machine learning (ML) models, affecting their performance and reliability. In response to this challenge, we present a microbenchmark study, called D3Bench, which evaluates the efficacy of open-source drift detection tools. D3Bench examines the capabilities of Evidently AI, NannyML, and Alibi-Detect, leveraging real-world data from two smart building use cases.We prioritize assessing the functional suitability of these tools to identify and analyze data drifts. Furthermore, we consider a comprehensive set of non-functional criteria, such as the integrability with ML pipelines, the adaptability to diverse data types, user-friendliness, computational efficiency, and resource demands. Our findings reveal that Evidently AI stands out for its general data drift detection, whereas NannyML excels at pinpointing the precise timing of shifts and evaluating their consequent effects on predictive accuracy.

The world and data are not static. But most machine learning models are. Once they are in production, they become less relevant with time. The data distributions evolve, the behavioral patterns change, and models need updates to keep up with new reality. The usual process is to retrain the models at defined intervals.

When we analyze machine learning model performance, we often focus on a single quality metric. With regression problems, this can be MAE, MAPE, RMSE, or whatever fits the problem domain best. Optimizing for a single metric absolutely makes sense during training experiments. This way, we can compare different model runs and can choose the best one. But when it comes to solving a real business problem and putting the model into production, we might need to know a bit more.

It is not easy to build a machine learning model. It is even harder to deploy a service in production. But even if you managed to stick all the pipelines together, things do not stop here. Once the model is in use, we immediately have to think about operating it smoothly. It is now delivering the business value, after all!