Fetching Query Results From Snowflake Just Got a Lot Faster With Apache Arrow

www.snowflake.com

We took our first step toward the adoption of Apache Arrow with the release of our latest JDBC and Python clients. Fetching result sets over these clients now leverages the Arrow columnar format to avoid the overhead previously associated with serializing and deserializing Snowflake data structures which are also in columnar format.

The Snowflake product team is jumping on the Arrow train! This is super-cool; it resulted in an up-to-10x performance improvement in some benchmarking the team did. This will likely impact data science use cases that get data out of Snowflake moreso than BI use cases—grabbing a full Pandas dataframe is way more data intensive than grabbing the aggregates that power a chart.

Read more...
Linkedin

Want to receive more content like this in your inbox?