The demand for large language model inference is rapidly increasing. Pipeline parallelism offers a cost-effective deployment strategy for distributed inference but suffers from high service latency. While incorporating speculative decoding to pipeline parallelism improves performance, it still faces challenges of low hardware utilization and narrow speculative window. Inspired by branch prediction in instruction pipelining, we introduce SpecPipe, which fills the pipeline with speculative tokens of a request step-by-step. By maximizing the hardware utilization, SpecPipe decodes one token per pipeline step ideally. Specifically, SpecPipe comprises a dynamic speculative token tree and a pipelined inference framework. The tree dynamically accepts tokens from a speculative token source and outputs the tokens to the inference pipeline. Since the speculative window relaxed in our framework, a high-accuracy draft model is integrated without fine-tuning. The pipeline inference framework follows node-wise computation, pruning propagation, and inter-node communication stages. We implement SpecPipe and a variant SpecPipe-DB with dynamic batching for single- and multi-request inference, respectively. On an 8-stage pipeline, SpecPipe improves time between tokens on diverse single-request workloads by $4.19\times$-$5.53\times$ over standard pipeline parallelism and by $2.08\times$-$2.38\times$ over prior tree-based speculative decoding methods. For multi-request workloads, SpecPipe-DB achieves $1.64\times$-$2.08\times$ higher throughput and $1.61\times$-$2.06\times$ lower time between tokens than vLLM.

In this article, you learn how to create and run machine learning pipelines by using the Azure Machine Learning SDK. Use ML pipelines to create a workflow that stitches together various ML phases. Then, publish that pipeline for later access or sharing with others. Track ML pipelines to see how your model is performing in the real world and to detect data drift. ML pipelines are ideal for batch scoring scenarios, using various computes, reusing steps instead of rerunning them, and sharing ML workflows with others. For guidance on creating your first pipeline, see Tutorial: Build an Azure Machine Learning pipeline for batch scoring or Use automated ML in an Azure Machine Learning pipeline in Python.

Figure 1 shows what we would like to have at the end of this article. In the following, we will implement each of these steps. In step 5, we apply hyperparameter optimization and create a feature importance plot. EDA, feature building, maximizing the model's performance, analyzing and interpreting the outcome are not in the scope of this article. The goal is to show you how to work with a pipeline that integrates modules from different packages.

What is Extreme Gradient Boosting?

When working on a machine learning project it is common to run numerous experiments in search of a combination of an algorithm, parameters and data preprocessing steps that would yield the best model for the task at hand. To keep track of these experiments Data Scientists used to log them into Excel sheets due to a lack of a better option. However, being mostly manual, this approach had its downsides. To name a few, it was error-prone, inconvenient, slow, and completely detached from the actual experiments. Luckily, over the last few years experiment tracking has come a long way and we have seen a number of tools appear on the market that improve the way experiments can be tracked, e.g.

Launched at AWS re:Invent 2020, Amazon SageMaker Pipelines is the first purpose-built, easy-to-use continuous integration and continuous delivery (CI/CD) service for machine learning (ML). With Pipelines, you can create, automate, and manage end-to-end ML workflows at scale. You can extend your pipelines to include steps for tasks performed outside of Amazon SageMaker by taking advantage of custom callback steps. This feature lets you include tasks that are performed using other AWS services, third parties, or tasks run outside AWS. Before the launch of this feature, steps within a pipeline were limited to the supported native SageMaker steps.

The issue with machine learning pipelines is that they need to pass state from one step to another. When this works, it's a beautiful thing to behold. When it doesn't, well, it's not pretty, and I think the clip below sums this up pretty well. Azure ML Pipelines are no stranger to this need for passing data between steps, so you have a variety of options at your disposal. This means it's not always easy to find the best one, and I've often seen people confused when trying to pick the best option.

The process of building and training Machine Learning models is always in the spotlight. There is a lot of talk about different Neural Network architectures, or new frameworks, facilitating the idea-to-implementation transition. While these are the heart of an ML engine, the circulatory system, which enables nutrients to move around and connects everything, is often missing. But what comprises this system? What gives the pipeline its pulse? The most important component in an ML pipeline works silently in the background and provides the glue that binds everything together.

Google today announced the beta launch of Cloud AI Platform Pipelines, a service designed to deploy robust, repeatable AI pipelines along with monitoring, auditing, version tracking, and reproducibility in the cloud. Google's pitching it as a way to deliver an "easy to install" secure execution environment for machine learning workflows, which could reduce the amount of time enterprises spend bringing products to production. "When you're just prototyping a machine learning model in a notebook, it can seem fairly straightforward. But when you need to start paying attention to the other pieces required to make a [machine learning] workflow sustainable and scalable, things become more complex," wrote Google product manager Anusha Ramesh and staff developer advocate Amy Unruh in a blog post. "A machine learning workflow can involve many steps with dependencies on each other, from data preparation and analysis, to training, to evaluation, to deployment, and more. It's hard to compose and track these processes in an ad-hoc manner -- for example, in a set of notebooks or scripts -- and things like auditing and reproducibility become increasingly problematic."

LazyGrid is a machine learning model comparator that follows the memoization paradigm, i.e. that is able to save fitted models and return them if required later. Lazygrid is supported on Python 3.5 and above. The package is compatible with scikit-learn 0.21 and Keras 2.2.5. In order to generate all possible pipelines given a set of steps, you should define a list of elements, which in turn are lists of pipeline steps, i.e. preprocessors, feature selectors, classifiers, etc. Each step could be either a sklearn object or a keras model.