Processing a Trillion Rows Per Second on a Single Machine: How Can Nested Loop Joins be this Fast?

I really think that understanding query optimizers is a super-important skillset, and one that is all-too-rare. This post delves into optimizer implementation in Spark and how it can produce novel and dramatic performance improvements. I learned a lot from this post.

The vast majority of big data SQL or MPP engines follow the Volcano iterator architecture that is inefficient for analytical workloads. Since Spark 2.0 release, the new Tungsten execution engine in Apache Spark implements whole-stage code generation, a technique inspired by modern compilers to collapse the entire query into a single function. This JIT compiler approach is a far superior architecture than the row-at-a-time processing or code generation model employed by other engines, making Spark one of the most efficient in the market.


Want to receive more content like this in your inbox?