Data drifts pose a critical challenge in the lifecycle of machine learning (ML) models, affecting their performance and reliability. In response to this challenge, we present a microbenchmark study, called D3Bench, which evaluates the efficacy of open-source drift detection tools. D3Bench examines the capabilities of Evidently AI, NannyML, and Alibi-Detect, leveraging real-world data from two smart building use cases.We prioritize assessing the functional suitability of these tools to identify and analyze data drifts. Furthermore, we consider a comprehensive set of non-functional criteria, such as the integrability with ML pipelines, the adaptability to diverse data types, user-friendliness, computational efficiency, and resource demands. Our findings reveal that Evidently AI stands out for its general data drift detection, whereas NannyML excels at pinpointing the precise timing of shifts and evaluating their consequent effects on predictive accuracy.

The explosion of data and its ever increasing complexity in the last few years, has made MLOps systems more prone to failure, and new tools need to be embedded in such systems to avoid such failure. In this demo, we will introduce crucial tools in the observability module of a MLOps system that target difficult issues like data drfit and model version control for optimum model selection. We believe integrating these features in our MLOps pipeline would go a long way in building a robust system immune to early stage ML system failures.

One of the important functions of MLOps engineers is to monitor the model performance. Data drift causes degradation in the model performance over a period of time. Let's discuss data drift and the steps we can take to detect it in detail. Data drift refers to changes in the data distribution over a period of time. Data drift can lead to poor model performance, because the model is being applied to data that is different from the data it was trained on.

In most big data analysis applications, data evolve over time and must be analyzed and treated in near real time. Patterns and interactions in such data often change over time, thus, models built for analyzing such data quickly become outdated over time. In machine learning and data mining this phenomenon is referred to as data drift. Data Drift Data drift is a change in the distribution of data over time. In machine learning models, data drift is the change in the distribution of a baseline data set on which the model was trained and the current real-time production data.

When data quality is fine, there are two usual suspects: data drift or concept drift. Data Drift -- The input data has changed. The distribution of the variables is meaningfully different. As a result, the trained model is not relevant for this new data. Concept Drift -- In contrast to the data drift, the distributions might even remain the same.

The most exciting moment of any machine learning system is when you get to deploy your model, but deploying becomes hard due to statistical issues such as "when past model performance is no more guaranteed for future and model performance degrade over a period of time due to changes of data when the model is deployed in a cloud with frequent data changes" and system engine such as system demands monitoring the ML system often which is manual in nature and tedious which needs to be handled through automation as much as possible. Now, How to deal with the statistical issue or degrading performance of the model?. How to handle the data changes once the model is deployed? That is where Concept and Data drift comes into the picture. Concept Drift refers to if the desired mapping from x to y changes and it leads to inaccurate predictions due to huge data distribution changes in the productized model.

MLOps isn't an algorithm, but it does operationalize the algorithm to simplify the predictive process,

How often have you heard "The Machine Learning Application worked well in the lab, but it failed in the field. It is not the fault of the Machine Learning Model! This blog is not yet another blog article (YABA) on DataOps, DevOps, MLOps, or CloudOps. I do not mean to imply xOps is not essential. For example, MLOps is both strategic and tactical. It promises to transform the "ad-hoc" delivery of Machine Learning applications into software engineering best practices. We know the symptoms: Most machine-learning models trained in the lab perform poorly on real-world data [1, 2, 3, 4]. Machine Learning created profits in the year 2020 and will continue to increase profits in the future. However, many problems hold back the progress and success of Machine Learning application rollout to production. I focus on what it is the most significant problem or cause: the quality and quantity of input data in Machine Learning models [1,4]. We realized the quantity of high-quality data was the bottleneck in predictive accuracy when we started showing near, or above, human-level performance in structured data, imagery, game playing, and natural language tasks. How many times do we look at the Machine Learning application lifecycle's conceptualization to realize a Machine Learning model is not at the beginning (Figure 2)? We can research and improve the tools of the Machine Learning application lifecycle. But that only lowers the cost of deployment. Arguably, the Machine Learning model's choice is not a critical part of deploying a Machine Learning application. We have a "good enough" process or pipeline to choose and change the Machine Learning model, given a training input dataset. However, when achieving State-of-the-Art (SOTA) results, the input data seems to have the most significant impact on the output predictive data (Figure 2). We seem to know the cause: input data that was garbage results in garbage output predictive data. New data input to a trained Machine Learning model determines the accuracy of the output. We divide Machine Learning input data into four arbitrary categories, defined by the Machine Learning application output accuracy. GPT-3 is an example [6]. GPT-3 trained with an enormous amount of data [6]. GPT-3 is frozen in time as a transformer that you access through an API. Concept Drift is a change in what to predict. For example, the definition of "what is a spammer." We do not cover Concept Drift here. I do not think of it as a problem but rather as a change in the solution's scope. An example of Case 2: Data Drift, is that Case 1: "It works!, is a temporal phenomenon.

In many real-world applications where machine learning models have been deployed in production, often the data evolve over time and thus models built for analyzing such data quickly become obsolete over time. It becomes essential for data scientists to monitor the model performance over time. Is the machine learning model deployed sustainable and performing consistently? Usually, the scenario that occurs over time, is not because the model stops performing well but simply because the model can no longer capture the right variability of the data be it the dependent or independent variables. The reason for this is not to do with the machine learning model itself, but the data distributions.

After obtaining a PhD in Biomedical Image Processing in 2011, Simona Maggio worked in several companies (CEA, Thales, Rakuten) as a Research Engineer in Computer Vision and Natural Language Processing for applications ranging from video surveillance to document digitization and e-commerce.