Designing Experimentation Guardrails

Introducing the Experiment Guardrails framework we implemented at Airbnb, which helps us prevent negative impact on key metrics while experimenting at scale.

Super-interesting. I'm always fascinated to hear about the experimentation systems implemented inside of big digital native tech companies because they are so far ahead of what everyone else has access to. This is the first time I've read about an automated monitoring system that looks at "guardrail" metrics to potentially end experiments early.

This is just like terminating a clinical trial early if you detect significant side effects of the treatment under study. Such a clearly good idea, but really hard to achieve in practice.


Want to receive more content like this in your inbox?