Successfully building and deploying a machine-learning model can be difficult to do once. Enabling other data scientists (or yourself) to reproduce your pipeline, compare the results of different versions, track what's running where, and redeploy and rollback updated models, is much harder. In this eBook, we'll explore in greater depth what makes the ML lifecycle so challenging compared to the traditional software-development lifecycle, and share the Databricks approach to addressing these challenges. Key challenges faced by organizations when managing ML models throughout their lifecycle and how to overcome them. How MLflow, an open source framework unveiled by Databricks, can help address these challenges, specifically around experiment tracking, project reproducibility, and model deployment.

Until now, the Machine Learning (ML) frameworks we've used at Borealis AI have varied according to individual preference. But as our applied team grows, we're finding that a preference-based system has certain shortcomings that have led to inefficiencies and delays in our research projects. As a result, we identified two main arguments in favour of standardizing a single framework for the lab. It has been our experience that independent frameworks do not often "play well" together. For example, a TensorFlow-based model applied to one research project would have to be rewritten in PyTorch for another project.

Now let's run the NbClust algorithm to estimate the ideal number of clusters (Figure 4). Figure 4. Number of clusters chosen by 26 indices, post standardization.