Predicting when Machine Learning Models Fail in Production - Naver Labs Europe

More crucially, this expensive maintenance process will continue forever as long as one would want a decent performance of their ML models that are deployed in production. Motivated by literature work from domain-shift and out of distribution detection,we propose a method that can predict the performance drop of a model when evaluated on a new target domain, without the need for any labelled examples from this target domain. Performing this estimation when done accurately and in real-time can have an important impact on the decision process of debugging and maintaining machine learning models in production. For instance, such insights can drive the decision to annotate more data for retraining or even adjusting models accordingly (e.g.