ETL Pipelines with Airflow: the Good, the Bad and the Ugly
Airflow is a popular open-source workflow management platform. Many data teams also use Airflow for their ETL pipelines. For example, I've previously used Airflow transfer operators to replicate data between databases, data lakes and data warehouses. I've also used Airflow transformation operators to preprocess data for machine learning algorithms. But is using Airflow for your ETL pipelines a good practice today?
Nov-20-2021, 04:57:00 GMT