A load balancer that learns, WebTorch – UnifyID – Medium

In my previous blog post "How I stopped worrying and embraced docker microservices" I talked about why Microservices are the bees knees for scaling Machine Learning in production. A fair amount of time has passed (almost a year ago, whoa) and it proved that building Deep Learning pipelines in production is a more complex, multi-aspect problem. Yes, microservices are an amazing tool, both for software reuse, distributed systems design, quick failure and recovery, yada yada. But what seems very obvious now, is that Machine Learning services are very stateful, and statefulness is a problem for horizontal scaling. An easy way to deal with this issue is understand that ML models are large, and thus should not be context switched.