Against A/B Tests

Michael Kaminsky is blowing up the current notion of an A/B test:

Traditional A/B testing rests on a fundamentally flawed premise. Most of the time, version A will be better for some subgroups, and version B will be better for others. Choosing either A or B is inherently inferior to choosing a targeted mix of A and B.

This is a fairly basic statement—it seems obviously true upon immediate consideration. It turns out to have rather significant impacts, though, if taken seriously. That is what the rest of the post explores.



Want to receive more content like this in your inbox?