Orchestrate Containerized Big Data Integration Jobs with Talend and Apache Airflow

In my last blog I described how to achieve continuous integration, delivery and deployment of Talend Jobs into Docker containers with Maven and Jenkins. This is a good start for reliably building your containerized jobs, but the journey doesn't end there. The next step to go further with containerized jobs is scheduling, orchestrating and monitoring them. While there are plenty of solutions you can take advantage of, I want to introduce an effective way to address this need for containerized Talend jobs in this blog. When it comes to data integration or even big data processing you need to go beyond simple task scheduling.