Converged Index™: The Secret Sauce Behind Rockset's Fast Queries

rockset.com

Learn how Rockset delivers low-latency SQL for search and analytics using a combination of row, column, and search indexes.

Ok, I've officially turned on to Rockset in the past couple of weeks. I've had the pleasure of meeting the CEO, Venkat, recently. He's impressive, the product is impressive, and it solves a real need. Here's the short version of why I think the product is interesting: it can be used as a serving layer for data products.

So...you've ingested a ton of data into Snowflake, you've build highly performant and modular transformations in dbt, and you've build a user interface that interacts with the final layer of transformed data. Here's the problem: almost certainly, your UI responsiveness is poor. Even if the data is modeled well, you'll see Snowflake's responsiveness typically at ~1 seconds on the low end. And for certain types of interactivity that number is higher. In an interactive context, users expect faster response times—typically more on the order of 50-200 ms—and so your product will always feel sluggish. This is one use case for Rockset. Take your final datasets and load them into Rockset, and voila!, you'll see interactive response times plummet. The post explains how they achieve this.

Designing a database engine is fundamentally about making tradeoffs, and Snowflake's core value proposition is crunching arbitrarily large datasets fast. In order to perform with well in other contexts, different tradeoffs need to be made. The Snowflake folks clearly realize this, which is why they're currently previewing their own search optimization service and query acceleration service. My guess is that they're attempting to have an in-platform answer to this exact problem. So much the better—it is a real pain point today.

Read more...
Linkedin

Want to receive more content like this in your inbox?