You now have the ability to run your Azure Machine Learning service pipelines as a step in your Azure Data Factory pipelines. This allows you to run your machine learning models with data from multiple sources (more than 85 data connectors supported in Data Factory). The seamless integration enables batch prediction scenarios such as identifying possible loan defaults, determining sentiment, and analyzing customer behavior patterns. Get started quickly by creating an AzureMLService connection and AzureMLExecutePipelne activity to invoke your Azure Machine Learning pipelines in a Data Factory data pipeline.
When you infuse AI (such as an ML.NET model) into your application, then your application lifecycle needs to be extended so it additionally embraces the'Machine Learning Model Lifecycle'. When deploying ML models to production, you need to automate the process to track, version, audit, certify and re-use every asset in your ML model lifecycle along with the end-user application lifecycle. In short, the ML model lifecycle process must be part of the application's Continuous Integration (CI) and Continuous Delivery (CD) pipelines. Let's walk through the diagram above to understand how this integration between the ML model lifecycle and the app development lifecycle can be achieved. For this common scenario, a starting assumption is that Git is used as your code repository, but it could be any other source code management platform.
Rosenbaum: This is the video of a machine-learning simulation learning to walk and facing obstacles, and it's there only because I like it. Also, it's a kind of metaphor for me trying to build the CI/CD pipeline. I'm going to be talking about CI/CD for machine learning, which is also being called MLOps. The words are hard, we don't have to really define these things, but we do have to define some other things and we're going to talk about definitions a lot actually. I'm going to start by introducing myself. I'm on the left, this picture is from DevOpsDays Chicago, our mascot is a DevOps Yak. You can come check out the conference. I work for Microsoft on the Azure DevOps team. I come from a developer background, and then, I did a lot of things with DevOps CI/CD and such. I'm not a data scientist, I did some classes on machine learning just so I can get context on this, but I'm coming to this primarily from a developer perspective. I also run another conference, this is a shameless plug, it's DeliveryConf, it's the first year it's happening, it's going to be in Seattle, Washington, on January 21 and 22. You should register for it right now because it's going to be awesome. The first thing I want to do is I want to set an agenda.
This reference architecture shows how to implement continuous integration (CI), continuous delivery (CD), and retraining pipeline for an AI application using Azure DevOps and Azure Machine Learning. The solution is built on the scikit-learn diabetes dataset but can be easily adapted for any AI scenario and other popular build systems such as Jenkins or Travis. A reference implementation for this architecture is available on GitHub. This build and test system is based on Azure DevOps and used for the build and release pipelines. Azure Pipelines breaks these pipelines into logical steps called tasks.
Scale out read-heavy workloads on Azure Database for PostgreSQL with read replicas, which enable continuous, asynchronous replication of data from one Azure Database for PostgreSQL master server to up to five Azure Database for PostgreSQL read replica servers in the same region. Replica servers are read-only except for writes replicated from data changes on the master. Stopping replication to a replica server causes it to become a standalone server that accepts reads and writes. Replicas are new servers that can be managed in similar ways as normal standalone Azure Database for PostgreSQL servers. For each read replica, you are billed for the provisioned compute in vCores and provisioned storage in GB/month.