A blog post describing how Spotify's event delivery has evolved after being on the Google Cloud Platform for two and a half years.
Heh. Wow. 65TB / day is a lot. Roughly 24PB / year at that pace, and the rate is continuing to grow.
I was very interested in Spotify's decision to heavily use GCP services—not just compute engine instances and GCS buckets, but services like PubSub, Bigquery, and many more—to build their pipeline on back in 2016. This decision was notably different from the decisions made at other consumer unicorns like Airbnb / Pinterest / Uber: those companies all use open source tools like Kafka and Presto as the foundation for their pipelines.
That's why this post is interesting: it's notably different than similar posts written by these other similar companies. Very useful perspective.Read more...