Just...wow. I really couldn't be a bigger fan of this post. It combines whole-company experimentation efforts + custom in-house data tech + multi-armed bandits, plus...data culture.
Really, this is such a fascinating seam for me. The hard part isn't data tooling or algorithms or experimentation... It's the systematic application of tooling and algorithms and experimentation within the operating system of the company. Building processes and technology and culture that actually enable successful experimentation at scale:
The idea is to have #oneway to run and analyze experiments across the entire business. The same platform is used by front-end engineers, back-end engineers, product managers, and data scientists. And it’s flexible enough to be used for experiments on inventory management and forecasting, warehouse operations, outfit recommendations, marketing, and everything in between.
So yeah, this post is kind of about MAB's (which are interesting in their own right) bit it's really about the massive difference between doing some cute A/B tests on your home page vs. enabling mass experimentation across your entire organization. The former might get you a short-term boost in conversion rate; the latter requires alignment and investment across your entire org but can create a long-term sustainable advantage.Read more...