Implementing an End-to-End Machine Learning Workflow with Azure Data Factory

The impression I had for implementing Machine Learning up to 3 years back was that of building a model in Python and deploying the project to an automated CI/CD pipeline. While it solved the basic criteria of performing predictions, it could never be called an end-to-end workflow because data storage and reporting were two significant components missing in this workflow and had to be dealt with separately. In this article, I will walk through an entire Machine Learning Operation Cycle and show how to establish every step of the way using Azure Data Factory (ADF). Yes, it is possible, easy, and extremely reliable. As a bonus, it also automatically sets you up to receive alerts for any sort of data anomalies occurring throughout the process, so you do not have to worry about monitoring the workflow manually.