Why Not Airflow?

medium.com

I've been following Prefect with interest since its open source release in March of this year, as have many members of the data community. Airflow is so prevalent as an orchestrator, and Prefect explicitly aims to build a next generation version of Airflow that many have taken notice (although with plenty of very natural skepticism!).

This post is the best answer yet to the questions "How is Prefect different than Airflow?" and "Why might I want to use Prefect instead of Airflow?"

Data engineering, and specifically in this case batch orchestration, is certainly not a solved problem. Airflow was massive step forwards for the entire industry, but it's been the dominant orchestration tool now for probably 4 years. If nothing else, this post represents smart people who have thought hard about what the next frontiers in batch orchestration will be.

Also re: Prefect—check out one of their cofounders, Jeremiah Lowin, on the Data Engineering Podcast last month. It was a good interview, Jeremiah is a dynamic speaker.

Read more...
Linkedin

Want to receive more content like this in your inbox?