A Comprehensive Guide On How to Monitor Your Models in Production - neptune.ai

Yup, that's me being plowed to the ground because the business just lost more than $500,000 with our fraud detection system by wrongly flagging fraudulent transactions as legitimate, and my boss's career is probably over. You're probably wondering how we got here… My story began with an image that you've probably seen over 1,001 times--the lifecycle of an ML project. A few months ago, we finally deployed to production after months of perfecting our model. I told myself and my colleague, "Our hard work has surely paid off, hasn't it?". Our model was serving requests in real-time and returning results in batches--good stuff! Surely that was enough, right? Well, not quite, which we got to realize in a relatively dramatic fashion. I'm not going to bore you with the cliché reasons why the typical way of deploying working software just doesn't cut it with machine learning applications. I'm still trying to recover from the bruises that my boss left on me, and the least I can do is help you not end up in a hospital bed after "successful model deployment", like me. I'll tell you all about: By the end of this article, you should know exactly what to do after deploying your model, including how to monitor your models in production, how to spot problems, how to troubleshoot, and how to approach the "life" of your model beyond monitoring. You almost don't have to worry about anything. Based on the software development lifecycle, it should work as expected because you have rigorously tested it and deployed it. In fact, your team may decide on a steady and periodic release of new versions as you mostly upgrade to meet new system requirements or new business needs.