Bauplan: zero-copy, scale-up FaaS for data pipelines

In this light, data workloads seem to Chaining functions for longer workloads is a key use case for FaaS be a natural fit for Function-as-a-Service (FaaS) platforms designed platforms in data applications. However, modern data pipelines to efficiently handle bursty, functional, and event-driven tasks. Unfortunately, differ significantly from typical serverless use cases (e.g., webhooks existing FaaS runtimes fall short in practice as they and microservices); this makes it difficult to retrofit existing pipeline were primarily designed to support the execution of many simple, frameworks due to structural constraints. In this paper, we describe independent functions that produce small outputs. Although popular these limitations in detail and introduce bauplan, a novel FaaS FaaS platforms (e.g., AWS Lambda [5], Azure Functions [17], and programming model and serverless runtime designed for data practitioners. OpenWhisk [4]) have added support for function chaining, their bauplan enables users to declaratively define functional capabilities fall short for data pipelines. It is therefore not surprising Directed Acyclic Graphs (DAGs) along with their runtime environments, that widely used data engineering frameworks (e.g., Airflow [1], which are then efficiently executed on cloud-based workers. Prefect [19], and Luigi [23]) lack native integration with serverless We show that bauplan achieves both better performance and a runtimes.