Auptimizer -- an Extensible, Open-Source Framework for Hyperparameter Tuning

--T uning machine learning models at scale, especially finding the right hyperparameter values, can be difficult and time-consuming. In addition to the computational effort required, this process also requires some ancillary efforts including engineering tasks (e.g., job scheduling) as well as more mundane tasks (e.g., keeping track of the various parameters and associated results). We present Auptimizer, a general Hyperparameter Optimization (HPO) framework to help data scientists speed up model tuning and bookkeeping. With Auptimizer, users can use all available computing resources in distributed settings for model training. The design also allows researchers to integrate new HPO algorithms. T o demonstrate its flexibility, we show how Auptimizer integrates a few major HPO techniques (from random search to neural architecture search). Designing a Machine Learning (ML) framework for production faces challenges similar to those faced with Big Data. There is a large volume of models with a variety of configurations and training them efficiently at scale with reproducibility is critical to realizing their business value. In this paper, we address one design aspect of the ML framework, namely the HPO process, via a framework called Auptimizer. A. Hyperparameter Optimization ML models are typically sensitive to the values of hy-perparameters [31]. Different from model parameters, these hyperparameters are values that control the model configuration or the training setup and thus need to be set before training the model. Due to the lack of gradient information for these hyperparameters, tuning them is often treated as a black-box optimization [11]. As an alternative to manual selection (which is usually based on modeler's expertise), researchers have proposed different methods to accelerate the tuning process including Bayesian approaches [26], evolutionary algorithms [10], multi-armed bandits [8], and architecture search by learning [33]. Tuning hyperparameters is often time-consuming especially when model training is computationally intensive [1]. Therefore, in practice, an automated HPO solution is critically important for machine learning. Both open-source solutions and commercial offerings are available.