Hey - I'm the head of engineering at Elementl, the company that builds Dagster. This post is my take on a crash-course introduction to Dagster. And if you want to support the Dagster Open Source project, be sure to star our Github repo. Dagster is a data orchestrator. Think of Dagster as a framework for building data pipelines, similar to how Django is a framework for building web apps.

This article is a rapid introduction to Dagster using a small ML project. It is beginner-friendly but might also suit more advanced programmers if they don't know Dagster. Data processing systems typically span multiple runtime, storage, tooling, and organizational boundaries. But all the stages in a data processing system share a fundamental property; they are directed acyclic graphs (DAGs) of functional computations that consume and produce data assets. Dagster is a data orchestrator for machine learning, analytics, and ETL.

Today the team at Elementl is proud to announce an early release of Dagster, an open-source library for building systems like ETL processes and ML pipelines. We believe they are, in reality, a single class of software system. We call them data applications. Dagster is a library for building these data applications. We define a data application as a graph of functional computations that produce and consume data assets.