Production Machine Learning: Determining ML Technical Debt

The intended audience for this post is both technical and non-technical stakeholders with the purpose of determining and explaining ML Technical Debt. Understanding of ML Technical Debt prevents various stakeholders, e.g., Project Manager, ML Engineers, Data scientists, Customers, and Investors from being blindsided by the excitement of a current/proof of concept ML product to only find out later that the ML product predictions are useless after a few months. Worse still, the expected final ML product never seems to come to a point of deployment value and the ML technical debt cannot be paid off due to financial and time constraints. Machine Learning (ML) Technical debt is the debt incurred by the deployment of an ML system without developing the code, infrastructure, tools and processes necessary for efficient iteration coupled with a lack of understanding and foresight of the actual ML product requirements. Provision of continuous and actual value for deployed ML systems is hindered by accumulated ML technical debt as the deployed ML system is unable to react to the changes and fails to achieve the necessary performance consistently.