This is both amazing and hilarious. I linked to a Kenny Ning post last year about an ML project he did @ Better, and the post got plenty of attention elsewhere. Turns out though, the work got very little usage internally. This is a reflection of what went wrong—why didn't users find the project valuable?
Here's my favorite section:
We probably didn’t need a fully productionized ML solution to improve our understanding of conversion. For example, consider this much simpler solution: a) Collect a dozen candidate features and fit a model offline. You can do this using a fancy ML library, but logistic regression in Excel works fine too. b) Pick the top 3 most predictive features and sense-check with a domain expert. c) Track those 3 features as KPIs in a line chart.
I think this is often a valuable (although less flashy) approach to data science work. Generate a novel insight using ML but then present the insight using traditional descriptive statistics.Read more...