The folks @ Elementl built a first-class dbt-Dagster integration! It's pretty great—I got a live demo from the data team at Good Eggs recently. In short, you can:
- Explicitly model the dependencies between your dbt models and processes that use other technologies;
- Schedule and execute pipelines that include both dbt and other technologies;
- Monitor your dbt models in the same tool you use to monitor your other processes, with historical and longitudinal views of your operations.
The monitoring part is particularly novel and useful IMO. Having great processes around metadata capture and analysis is really the next level of maturity of the dbt workflow—with that information, you can direct your attention as a maintainer so much more effectively.Read more...