End-to-end Machine Learning Pipeline with Docker and Apache Airflow from scratch

This post describes the implementation of a sample Machine Learning pipeline on Apache Airflow with Docker, covering all the steps required to setup a working local environment from scratch. Let us imagine to have a Jupyter Notebook with a polished Machine Learning experiment, including all the stages that lead from raw data to a fairly performant model. In our scenario, new input data is provided by daily batches, and the training procedure should be performed as soon as a new batch is provisioned, in order to tune the model's parameters to accomodate data changes. Moreover, experiment's parameters, training conditions and performances should be tracked with the aim to monitor the results of the different training sessions. Finally, the obtained models should be saved and made available to other systems to be used for inference, allowing, at the same time, version control over each generated model.