Excellent writeup of a sophisticated experimentation methodology in a large network.
Given the systemic nature of many of our products, simple A/B tests are often ineffective due to network effects. To be able to experiment in the face of network effects, we use a technique known as switchback testing, where we switch back and forth between treatment and control in particular regions over time.
Fun fact: switchback testing was originally employed in an agricultural context, specifically for cow lactation experiments.
This reminds me of a post that Lyft wrote a while ago on their experimentation methodology. It turns out that naive A/B testing is insufficient in more complicated experimental environments.Read more...