Logs & Offsets: (Near) Real Time ELT with Apache Kafka + Snowflake

medium.com

Wowowow. Convoy has implemented a near-real-time ELT pipeline across their entire data stack while staying SQL-first in the transformation layer. It's Snowflake's streams and tasks that enable the real-timiness in the T layer, while Debezium + Kafka handle the data ingestion (they had to move off of an unnamed managed ingestion service due to increasing latency as dataset sizes grew).

I've written a lot over the years about the modern ELT stack and this is the first post where I've seen a meaningful improvement to the reference infrastructure we've been building for clients since 2016.

Read more...
Linkedin

Want to receive more content like this in your inbox?