Continuous Delivery for Machine Learning

In the famous Google paper published by Sculley et al. in 2015 "Hidden Technical Debt in Machine Learning Systems", they highlight that in real-world Machine Learning (ML) systems, only a small fraction is comprised of actual ML code. There is a vast array of surrounding infrastructure and processes to support their evolution. They also discuss the many sources of technical debt that can accumulate in such systems, some of which are related to data dependencies, model complexity, reproducibility, testing, monitoring, and dealing with changes in the external world. Many of the same concerns are also present in traditional software systems, and Continuous Delivery has been the approach to bring automation, quality, and discipline to create a reliable and repeatable process to release software into production. "Continuous Delivery is the ability to get changes of all types -- including new features, configuration changes, bug fixes, and experiments -- into production, or into the hands of users, safely and quickly in a sustainable way".