Artificial Intelligence (AI) has massive potential to drive disruptive innovations affecting most enterprises on the planet. However, most enterprises are struggling to succeed with AI . Simply put, AI and Data are siloed in different systems and different organizations. Enterprise data is siloed across hundreds of systems such as data warehouses, data lakes, databases and file systems that are not AI-enabled. Popular machine learning frameworks such as TensorFlow, PyTorch, and SciKit-Learn don't do data processing.

Successfully building and deploying a machine-learning model can be difficult to do once. Enabling other data scientists (or yourself) to reproduce your pipeline, compare the results of different versions, track what's running where, and redeploy and rollback updated models, is much harder. In this eBook, we'll explore in greater depth what makes the ML lifecycle so challenging compared to the traditional software-development lifecycle, and share the Databricks approach to addressing these challenges. Key challenges faced by organizations when managing ML models throughout their lifecycle and how to overcome them. How MLflow, an open source framework unveiled by Databricks, can help address these challenges, specifically around experiment tracking, project reproducibility, and model deployment.

Maintaining assets such as compressors is an extremely complex endeavor: they are used in everything from small drilling rigs to deep-water platforms, the assets are located across the globe, and they generate terabytes of data daily. A failure for just one of these compressors costs millions of dollars of lost production per day. An important way to save time and money is to use machine learning to predict outages and issue maintenance work orders before the failure occurs. Ultimately, you need to build an end-to-end predictive data pipeline that can provide a real-time database to maintain asset parts and sensor mappings, support a continuous application that processes a massive amount of telemetry, and allows you to predict compressor failures against these datasets. Our approach to addressing these issues is by selecting a unified platform that offers these capabilities.

On July 9th, our team hosted a live webinar--Scalable End-to-End Deep Learning using TensorFlow and Databricks--with Brooke Wenig, Data Science Solutions Consultant at Databricks and Sid Murching, Software Engineer at Databricks. In this webinar, we walked you through how to use TensorFlow and Horovod (an open-source library from Uber to simplify distributed model training) on the Databricks Unified Analytics Platform to build a more effective recommendation system at scale. If you missed the webinar, you can view it now as well download the slides here. If you'd like free access Databricks Unified Analytics Platform and try our notebooks on it, you can access a free trial here. Toward the end, we held a Q&A, and below are all the questions and their answers.