Kubeflow Emerges for ML Workflow Automation

Many data scientists today find it burdensome to manually execute all of the steps in a machine learning workflow. Moving and transforming data, training models, then promoting them into production – all of it requires the data scientist's close attention. But now an open source project called Kubeflow promises to eliminate much of that busywork by automating machine learning workflows atop Kubernetes clusters. Google initially created Kubeflow to manage its internal machine learning pipelines written in Tensorflow and executed atop Kubernetes, and released it as an open source project in late 2017. Since then, the Kubeflow community has integrated the software with a handful of additional machine learning and deep learning frameworks, including MXnet, PyTorch, Caffe2, and Nvidia TensorRT, as well as Jupyter notebooks and MPI, the parallel computing framework used in high performance computing (HPC) clusters.