Too Long; Didn't Read There are a lot of Machine Learning courses, and we are pretty good at modeling and improving our accuracy or other metrics. But a lot of us are getting in trouble outside the Jupyter/VS Code. There is a gap between our models and finalized business solution. And it doesn't matter how good our models are if they don't create value for the business. Finally, it is satisfying to have a fully working solution.

Follow along with our new YOLOv5 deployment tutorial to power your next object detection application. Or, watch this tutorial video by Smitha Kolan on how to deploy YOLOV5 in under 15 minutes using the OctoML CLI. Today, we are excited to announce the results of our collaboration with Ultralytics to deploy the YOLOv5 models to over 100 CPU and GPU hardware targets in AWS, Azure and GCP. Our engineering work with Ultralytics unlocks the ability to deploy YOLOv5 models on hardware from Intel, NVIDIA, Arm and AWS, with minimal effort and cost. In this blog, I'll show you how simple it is to achieve hardware independence and cost savings across multiple clouds.

Large language models (LLMs) are some of the most advanced deep learning algorithms that are capable of understanding written language. Many modern LLMs are built using the transformer network introduced by Google in 2017 in the Attention Is All You Need research paper. NVIDIA NeMo Megatron is an end-to-end GPU-accelerated framework for training and deploying transformer-based LLMs up to a trillion parameters. In September 2022, NVIDIA announced that NeMo Megatron is now available in Open Beta, allowing you to train and deploy LLMs using your own data. With this announcement, several pretrained checkpoints have been uploaded to HuggingFace, enabling anyone to deploy LLMs locally using GPUs.

Model deployment in machine learning (ML) is becoming increasingly complex. You want to deploy not just one ML model but large groups of ML models represented as ensemble workflows. These workflows are comprised of multiple ML models. Productionizing these ML models is challenging because you need to adhere to various performance and latency requirements. Amazon SageMaker supports single-instance ensembles with Triton Inference Server.

In this tutorial, we will configure and deploy Nvidia Triton Inference Server on the Jetson Mate carrier board to perform inference of computer vision models. It builds on our previous post where I introduced Jetson Mate from Seeed Studio to run the Kubernetes cluster at the edge. Though this tutorial focuses on Jetson Mate, you can use one or more Jetson Nano Developer Kits connected to a network switch to run the Kubernetes cluster. Assuming you have installed and configured JetPack 4.6.x on all the four Jetson Nano 4GB modules, let's start with the installation of K3s. The first step is to turn Nvidia Container Toolkit into the default runtime for Docker.

Containers have become the unit of deployment not just for data center and cloud workloads but also for edge applications. Along with containers, Kubernetes has become the foundation of the infrastructure. Distributions such as K3s are fueling the adoption of Kubernetes at the edge. I have seen many challenges when working with large retailers and system integrators rolling out Kubernetes-based edge infrastructure. One of them is the ability to mix and match ARM64 and AMD64 devices to run AI workloads.

If you are familiar with Kubeflow, you know KFServing as the platform's model server and inference engine. In September last year, the KFServing project has gone through a transformation to become KServe. KServe is now an independent component graduating from the Kubeflow project, apart from the name change. The separation allows KServe to evolve as a separate, cloud native inference engine deployed as a standalone model server. Of course, it will continue to have tight integration with Kubeflow, but they would be treated and maintained as independent open source projects.

At a Glance: So far we have gone through the pre-processing pipeline of the Criteo Dataset using the standard NVTabular toolkit present inside Merlin SDKs. This was a part of the ETL processing and it is present under Part 1. Followed by this, in Part 2, we have gone through the training procedures using the HugeCTR architecture. We have explored 4 standard state-of-the-art architectures using HugeCTR training toolkits inside Merlin. In this section, we are going to explore the inference implementation which is the final deployment process using the Triton Inference Server. Nvidia's Merlin contains 3 crucial components.

Nvidia has rolled out the latest version of its AI Enterprise suite for GPU-accelerated workloads, adding integration for VMware's vSphere with Tanzu to enable organisations to run workloads in both containers and inside virtual machines. Available now, Nvidia AI Enterprise 1.1 is an updated release of the suite that GPUzilla delivered last year in collaboration with VMware. It is essentially a collection of enterprise-grade AI tools and frameworks certified and supported by Nvidia to help organisations develop and operate a range of AI applications. That's so long as those organisations are running VMware, of course, which a great many enterprises still use in order to manage virtual machines across their environment, but many also do not. However, as noted by Gary Chen, research director for Software Defined Compute at IDC, deploying AI workloads is a complex task requiring orchestration across many layers of infrastructure.

NVIDIA opened the door for enterprises worldwide to develop and deploy large language models (LLM) by enabling them to build their own domain-specific chatbots, personal assistants and other AI applications that understand language with unprecedented levels of subtlety and nuance. The company unveiled the NVIDIA NeMo Megatron framework for training language models with trillions of parameters, the Megatron 530B customizable LLM that can be trained for new domains and languages, and NVIDIA Triton Inference Server with multi-GPU, multinode distributed inference functionality. Combined with NVIDIA DGX systems, these tools provide a production-ready, enterprise-grade solution to simplify the development and deployment of large language models. "Large language models have proven to be flexible and capable, able to answer deep domain questions, translate languages, comprehend and summarize documents, write stories and compute programs, all without specialized training or supervision," said Bryan Catanzaro, vice president of Applied Deep Learning Research at NVIDIA. "Building large language models for new languages and domains is likely the largest supercomputing application yet, and now these capabilities are within reach for the world's enterprises."