Etsy: The Causal Analysis of Cannibalization in Online Products

Heh. A/B testing is hard. Etsy recently realized that an improved recommendation algorithm actually hurt overall site-wide conversion rate and GMV! This turned out to be because this improvement actually cannibalized users' engagement with search. This (rather dense) post goes through how they proved the magnitude of this effect.

For me, this is another nail in the coffin of the "let's just throw [A/B testing tool of choice] on the site and run some experiments!" approach to experimentation. Unless you're ready to think deeply about your experimental strategy and spend real time analyzing the results, tooling isn't going to produce sustainable outcomes on its own.


Want to receive more content like this in your inbox?