Distributed Hyperparameter Tuning in Vertex AI Pipeline

Vertex AI pipelines offer a handy way to implement end-to-end ML workflows from data collection to endpoint monitoring with extremely low effort. For new users, the easiness of development and deployment is largely thanks to the Vertex AI pipeline example offered by GCP. Despite the comprehensive demonstration of the essential components, the official example also exposes the feasibility for users to customize and enhance the pipeline based on their own needs. Amongst all, one of the most exciting components is the distributed Hyperparameter Tuning (HPT) that is capable of exploring a huge search space to identify the best hyperparameters in a short time. However, the limitation of the tutorials is that the distributed HPT is presented as a standalone HPT task/pipeline and it doesn't explicitly present the approach to integrate into the existing Vertex AI pipeline shown in the Vertex AI pipeline example, which motivates me to share my successful attempt that bridges the gap. I believe this will benefit many businesses who have built or will build their ML workflows based on Vertex AI pipeline.