Spotify has open-sourced their Terraform module for running machine-learning pipeline software Kubeflow on Google Kubernetes Engine (GKE). By switching their in-house ML platform to Kubeflow, Spotify engineers have achieved faster time to production and are producing 7x more experiments than on the previous platform. In a recent blog post, Spotify's product manager Josh Baer and ML engineer Samuel Ngahane described Spotify's "Paved Road" for machine learning: "an opinionated set of products and configurations to deploy an end-to-end machine learning solution using our recommended infrastructure." By adopting these standards, Spotify's machine learning engineers no longer need to build or maintain infrastructure and instead can focus on their ML experiments. Since launching the platform in mid-2019, about 100 internal users have adopted it and run up to 18,000 experiments.

As a result, our customers rarely have to build their own Docker containers, once again focusing less on infrastructure and more on their core use case. By having this controlled layer between the user and Kubeflow, we can easily manage upgrades of Kubeflow and TFX. We launched the alpha version of our platform in August and so far we have already seen about 100 users totaling 18,000 runs. Machine learning engineers can now focus on designing and analyzing their ML experiments instead of building and maintaining their own infrastructure, resulting in faster time from prototyping to production. In fact, early analysis indicates some teams are producing 7x more experiments already!