Unbundling Data Science Workflows with Metaflow and AWS Step Functions

We believe that problems are solved by people, not by tools. Following our human-centric, usability-driven approach, data scientists shouldn't have to care about the lower layers of the stack -- they should just work -- but we believe that there is no benefit in trying to pretend that the stack doesn't exist, which would be problematic especially when things fail. This article focuses on the job scheduler layer and the two layers that surround it: The architecture layer that defines the structure of the user's code, and the compute layer that defines how the code is executed. Since the initial open-source release of Metaflow, we have heard questions about how Metaflow compares to other workflow schedulers or how Metaflow workflows should be executed in production. The answer to both of these questions is the same: Metaflow is designed to be used in conjunction with a production-grade job scheduler. Today, we are releasing the first open-source integration with such a scheduler, AWS Step Functions, which you can use to execute your Metaflow workflows in a scalable and highly-available manner. Before going into details about AWS Step Functions, we want to highlight the role of the job scheduling layer in the Metaflow stack.