How Etsy Handles Peeking in A/B Testing

In this post, we investigate (...) how to peek at experimental results early in order to increase the velocity of our decision-making without sacrificing the integrity of our results.

Have you ever run an A/B test? Have you ever seen the lines in your A/B testing tool cross over the threshold and wanted to immediately stop the test and declare victory? I have. It's so tempting—you desperately want your hypothesis to be right!

Looking at the results before the experiment is over—peeking—can ruin the statistical basis of your experiment if you're not thoughtful about how you do it. This article goes into how Etsy enables peeking (there are real reasons you might want to end an experiment early!). They've designed this into their internal A/B testing tool and show screenshots.

Very interesting—new to me.


Want to receive more content like this in your inbox?