The Dawn of Zendesk's Machine Learning Model Building Platform with AWS Batch

When we worked on Content Cues, one of Zendesk's machine learning products, we encountered the scalability challenge of having to build up to 50k machine learning (ML) models daily. Looking at the data was initially nerve-wracking. This article focuses on the new model building platform we designed and built for Content Cues, and has been running on AWS Batch in production for a few months. From conception to implementation, the process has been a challenging yet rewarding experience for us, and we would like to share our journey with you. This is the first of a 3 part series, covering how we evaluated different technology options (AWS Batch, AWS Sagemaker, Kubernetes, EMR Hadoop/Spark), ultimately deciding on AWS Batch.