The Machine Learning Reproducibility Crisis

petewarden.com

I was recently chatting to a friend whose startup’s machine learning models were so disorganized it was causing serious problems as his team tried to build on each other’s work and share it with clients. Even the original author sometimes couldn’t train the same model and get similar results! He was hoping that I had a solution I could recommend, but I had to admit that I struggle with the same problems in my own work. It’s hard to explain to people who haven’t worked with machine learning, but we’re still back in the dark ages when it comes to tracking changes and rebuilding models from scratch. It’s so bad it sometimes feels like stepping back in time to when we coded without source control.

100% agree. We may have made a lot of progress in the past couple of years deploying ML systems, but the industry is nowhere near a point of process maturity. We still suck at this.

Read more...
Linkedin

Want to receive more content like this in your inbox?