The Materialize Incremental View Maintenance Engine

www.datacouncil.ai

...

This is definitely the most exciting thing I've come across in the past little while. I spoke to the CEO this week and got the download on it, and have also watched the talk from the link above, given by their Chief Scientist.

Materialize is a streaming SQL materialized view product. It competes directly with KSQL, but has some important relative technical and UX benefits. It’s all based on long-running academic research (since 2013) and an open-source project called Timely Dataflow.

IMO, the main benefit vs. something like KSQL is that it handles joins seamlessly. It can handle the entire set of TPC-H queries, including 8-way joins, and can keep multiple nested layers of materialized views up-to-date within milliseconds. Its data is stored in a columnar format under-the-hood, so it's optimized for analytic workloads.

While the research has been ongoing for a while, the commercial product is only just now being released. Beta is Jan/Feb. If you have use cases for streaming SQL processing, I'd highly recommend watching this full talk and staying in touch with Materialize as the launch happens.

We plan on experimenting with building a dbt adapter as soon as we can get our hands on an early release. If Materialize can deliver on the demo shown in this talk, it unlocks a bunch more use cases for data pipelines.

Read more...
Linkedin

Want to receive more content like this in your inbox?