A/B Testing at Stack Overflow


Julia Silge writes a nuanced, realistic take on A/B testing at Stack Overflow. My favorite paragraph comes at the end of the post:

Sometimes the results of an A/B test can be inconclusive, with no measurable difference between the baseline and new version, either positive or negative. What should we do then? Often we stay with the original version of our feature, but in some situations, we still decide to make a change to a new version, depending on other product considerations.

My experiences in running A/B tests are similar: the clear-cut cases are easy, but it's hard when you're running a business and the test outcome isn't definitive. This isn't covered in your stats textbook.


Want to receive more content like this in your inbox?