Building a data processing pipeline is one of the most common problem statements, for which you would have written small scripts or built a full-fledged scalable system based on the amount, and frequency of data. In this article, we will talk about the idea of event-driven scalability, the backbone that will be cost-optimized, and requires a minimum amount of development and operations. Why build an event-driven and scalable data processing pipeline? While working with startups or building a team project or a personal project, that requires a pipeline for data processing, there is always a constraint of cost. We will use a simple example for building a metadata extraction system for e-commerce products.

Google Cloud Run became Generally-Available (GA) around November of 2019. It provides a fully managed serverless execution platform to abstract infrastructure for stateless code deployment with HTTP-driven containers. Cloud Run is a Knative service utilizing the same APIs and runtime environments that make it possible to build container-based applications that can run anywhere, whether on Google cloud or Anthos deployed on-premises or on the Cloud. As a "serverless execution environment", Cloud Run can scale in response to the computing needs of the running application. Instant execution of application code, scalability and portability are core features of Cloud Run.

A lot of people were interested in the architecture behind Cartoonizer. So Tejas and I (Niraj) have tried explaining the process we employed to make it work. In today's fast paced world, ML experts are expected to put on multiple hats in the ML workflow. One of the critical tasks in the workflow is to serve the models in production! This seemingly important piece in the pipeline tends to get overlooked thus faltering to provide value to the customers.

One of the main challenges that every data scientist face is model deployment. Unless you are one of the lucky few who has loads of data engineers to help you deploy a model, it's really an issue in enterprise projects. I am not even implying that the model needs to be production ready but even a seemingly basic issue of making the model and insights accessible to business users is more of a hassle then it needs to be. These are two ends of the spectrum. Ad-hoc runs are just too tedious and clients typically demand for some self-serve interface but good luck trying to get a permanent server to host your code.