Great post exploring the often neglected problem of technical debt in machine learning systems. The author presents three types of technical debt:
- Feedback Loops: you're ML system is fed data that it generated itself, improving performance metrics without actually improving the system. Fix lies in proper exploitation / exploration calibration.
- Correction Cascades: If you apply too many fixes and heuristic corrections to your ML system as a result you are no linger able to properly train your system as a whole.
- Hobo-Features: useless features that are hard to get rid of, e.g. a feature that gave a minor performance boost but has become neutral once more data was collected.