Prefect: Positive and Negative Data Engineering

medium.com

One of the leaders of the Apache Airflow project has been working on a (potentially?) new and improved data engineering platform called Prefect. Its focus: detect when pipeline issues arise without forcing you to write a million conditionals to detect every possible error scenario.

The post is interesting, but the tutorial doesn't really talk about how that secret sauce actually happens. I'm a little skeptical at this juncture, but at the same time have a lot of respect for the author's background in the problem space. I'd recommend following Prefect if you're an Airflow user today.

Read more...
Linkedin

Want to receive more content like this in your inbox?