Deploying ML models in Production and scaling your ML services still continue to be big challenge. TorchServe, the model serving solution for PyTorch solves this problem and has now evolved into a multi-platform solution that can run on-prem or on any cloud with integrations for major OSS platforms like Kubernetes, MLflow, Kubeflow Pipelines, KServe. This talk will cover new features launched in TorchServe like model interpretability using Captum, best practices for production deployments in a responsible manner, along with examples of how companies like Amazon Ads, Meta AI and broader PyTorch community are using TorchServe.

BentoML is a flexible, high-performance framework for serving, managing, and deploying machine learning models. By providing a standard interface for describing a prediction service, BentoML abstracts away how to run model inference efficiently and how model serving workloads can integrate with cloud infrastructures. Be sure to check out deployment overview doc to understand which deployment option is best suited for your use case. BentoML provides APIs for defining a prediction service, a servable model so to speak, which includes the trained ML model itself, plus its pre-processing, post-processing code, input/output specifications and dependencies. The generated BentoML bundle is a file directory that contains all the code files, serialized models, and configs required for reproducing this prediction service for inference. BentoML automatically captures all the python dependencies information and have everything versioned and managed together in one place.

As more and more industries bring ML use cases to production, the need for consistent practices for managing ML in Production and optimizing ML Lifecycle iteration has grown rapidly. Last year, a few of us partnered with USENIX to drive the first-ever Industry/Academic conference dedicated to the challenges of and innovations in managing ML in Production. OpML 2019 was a great success - bringing together experts, practitioners, engineers, and researchers to discuss the latest and greatest in ML Ops. You can find a summary of OpML 2019 here. This year, due to COVID19, OpML 2020 became a virtual conference with video presentations and open discussions on Slack.