Databricks End-To-End Machine Learning - Create An Ingest-To-Serving MLOps Pipeline

Create a Python notebook in your Databricks workspace and attach it to a suitable Databricks ML cluster. I'm only using Scala in the following steps because it appeared to be the easiest way to get the data from the public CDC URL into a Spark dataframe without having to download files locally. This can be done in a Databricks Python notebook by using the Scala magic command %scala at the top of each cell with Scala code. All the actual Machine Learning code later on will be written in Python. I am going to use Databricks AutoML in the next step which does its own training/evaluation/test split so the above is mainly to have some data for testing (holdout) the best AutoML model after it has been created on data that the AutoML process has not seen at all yet.