Scaling Apache Airflow for Machine Learning Workflows

Apache Airflow is a popular platform to create, schedule and monitor workflows in Python. It has more than 15k stars on Github and it's used by data engineers at companies of all sizes including Twitter, Airbnb and Spotify. If you're using Apache Airflow, your architecture has probably evolved based on the number of tasks and their requirements. While working at Skillup, we first had a few hundred DAGs to execute all our data engineering tasks. Then we started doing machine learning.