Visually Inspecting Data Profiles for Data Distribution Shifts

The null hypothesis is that the samples are drawn from the same distribution, which means that a low p-value is indicative of different distributions. In this example, we see that drift was detected for all of our features. In addition to statistical tests, there are other approaches you can take to tackle distribution shifts, such as visually inspecting histograms and distribution charts for individual features, which can be useful to confirm the disparity between distributions. In a more general topic, setting rule-based data validation is key in ensuring the quality of your data, which includes distribution changes, be it from external factors or systemic errors such as pipeline errors or missing data. For a more in-depth view on this topic, you can sign up for my upcoming workshop at ODSC Europe this June "Visually Inspecting Data Profiles for Data Distribution Shifts". In the workshop, we will also see how to visually inspect histograms and distribution charts and how to do data validation with whylogs' constraints. We will dig deeper into the concept of distribution shift and explore other popular packages in order to detect data shifts.