It's moved from an academic discipline to one of the most exciting technologies around. From understanding video feeds in self-driving cars to personalizing medications, it's becoming important in every industry. While the model architectures and concepts have received a lot of attention, machine learning has yet to go through the standardization of processes that the software industry experienced in the last two decades. In this book, we'd like to show you how to build a standardized machine learning system that is automated and results in models that are reproducible. Who Is This Book For?

You can check the code for this tutorial here. Once you finish your model experimentation it is time to roll things to production. Rolling Machine Learning to production is not just a question of wrapping the model binaries with a REST API and starting to serve it, but and making it possible to re-create (or update) and re-deploy your model. That means the steps from preprocessing data to training the model to roll it to production (we call this a Machine Learning Pipeline) should be deployed and able to be run as easily as possible while making it possible to track it and parameterize it (to use different data, for example). In this post, we will see how to build a Machine Learning Pipeline for a Deep Learning model using Tensorflow Extended (TFx), how to run and deploy it to Google Vertex AI and why should we use it.

Oftentimes in machine learning, there is a confusion about how to build a scalable and robust models which can be deployed in real-time. The thing that mostly complicates this is the lack of knowledge about the overall workflow in machine learning. Understanding the various steps in machine learning workflow can be especially handy for data scientists or machine learning engineers as it saves a considerable amount of time and effort in the long run. In this article, we will be going over the steps that are usually involved in building a machine learning system. Having a good understanding of the principles needed to build a high-level design of an AI system is useful so that one could allocate their time and resources to complete each part of the puzzle before coming up with a robust high-performance model that is put to production.

In recent years, there have been rapid advancements in Machine Learning and this has led to many companies and startups delving into the field without understanding the pitfalls. Common examples are the pitfalls involved when building ML pipelines. Machine Learning pipelines are complex and there are several ways they can fail or be misused. Stakeholders involved in ML projects need to understand how Machine Learning pipelines can fail, possible pitfalls, and how to avoid such pitfalls. There are several pitfalls you should be aware of when building machine learning pipelines. The most common pitfall is the black-box problem -- where the pipeline is too complex to understand.

Let's go a step further and say we don't even want to select the columns in advance. Instead, we need the pipeline to do the selection for us. Now, we introduce FeatureUnion- It concatenates the results of multiple transformations happening in parallel! Since pipeline wants classes as selectors that have fit, transform methods, we will extend the base classes BaseEstimator & TransformerMixin. We could have applied grid-search, random search, and cross-validations in all of them!

But, what does that mean? In a crux, they help develop the sequential flow of data from one estimator/transformer to the next till it reaches the final prediction algorithm. It ensures there is no data leakage between train, test, and validation sets. The pipeline also makes a program more automated to be used as a functional code. We will use grid search to tune hyperparameters and generate output.