This post is so timely for me. As dbt's transformation capabilities become increasingly mature, we continue to be very focused on understanding the adjacent problems, many of which can be summed up under the category of metadata. And as usual, Airbnb is living in the future.
This post describes the tool that a team at Airbnb built to track and analyze timeliness and SLA compliance for their various datasets. The author describes some really meaty problems: the implicit understanding of the DAG required to answer the question "why was this dataset late?" The criticality of locating bottlenecks. More.
The entire ecosystem will have this type of tooling, and soon.Read more...