Introducing Dagster

medium.com

Today the team at Elementl is proud to announce an early release of Dagster, an open-source library for building systems like ETL processes and ML pipelines.

I linked to an article about Prefect in the last issue; here's a similar post from competitive product, Dagster. The post itself is solid, but what I think is particularly interesting here is that this "next generation orchestration solution" space seems to have gone from cold to hot in the space of about a month. Word on the street is that both Prefect and Elementl (which makes Dagster) are both closing (or have closed?) sizable venture rounds within just a few weeks of each other.

As a result, there will be a lot of innovation in this space over the coming years, which is exciting: Airflow was a huge step forwards, but it's been several years since that paradigm shift and our data engineering challenges are very much not solved. I'm excited to watch this space evolve.

Read more...
Linkedin

Want to receive more content like this in your inbox?