There's so much to like here. The meat of the argument can be summed up here:
There is no correct win rate waiting to be unearthed; one version isn’t true while another is false. Each version is equally accurate because they are tautological: They measure precisely what they say they measure, no more and no less. Our job as analysts isn’t to do the math right so that we can figure out which answer is in the back of the book; it’s to determine which version, out of a subjective set of options, helps us best run a business.
This gets so effectively at the thing that I've found to be the single hardest thing to mentor other data analysts on: business context! What is interesting? What should make you curious to dig in deeper? What change would a given piece of information lead you to make in the world?
Another thing I've been thinking about recently is explicitly creating separate workspaces for curated (production) and messy (experimental, not-yet-production) work. I think one of the challenges Benn is describing isn't just that analytics is messy...it's that teams often co-mingle the messy with the clean parts of the process.
Which ends up coming down to environment management. Creating end-to-end workflows that facilitate environment management is harder than it should be today. How could we make that easier as an ecosystem?Read more...