Improving a Machine Learning System (Part 1 - Broken Abstractions)

This post is part one in a three part series on the challenges of improving a production machine learning system. Find part two here and part three here. Suppose you have been hired to apply state of the art machine learning technology to improve the Foo vs Bar classifier at FooBar International. Foo vs Bar classification is a critical business need for FooBar International, and the company has been using a simple system based on a decade-old machine learning technology to solve this problem for the last several years. As a machine learning expert, you are shocked that FooBar International hasn't gotten around to modernizing this system, and you are confident that replacing the Foo vs Bar classifier with the latest machine learning hotness will dramatically improve system performance.