Key Challenges with Quasi Experiments at Netflix

We face various business problems where we cannot run individual level A/B tests but can benefit from quasi experiments. For instance, consider the case where we want to measure the impact of TV or billboard advertising on member engagement. It is impossible for us to have identical treatment and control groups at the member level as we cannot hold back individuals from such forms of advertising.

This practice is one I understand well, but I actually didn't realize that it had a name! Quasi-experiments. Shopify has written about their approach to quasi-experimentation as well. Read together, these posts present and excellent roadmap for these types of experimental challenges.


Want to receive more content like this in your inbox?