Bayesian Optimization Using Monotonicity Information and Its Application in Machine Learning Hyperparameter

Bayesian optimization has been successfully applied to many global optimization problems (Jones et al., 1998; Martinez-Cantin et al., 2007; Hutter et al., 2011; Snoek et al., 2012). Typically, it makes few assumptions about the objective function, treating it as a black box. When prior knowledge is available, however, it might be possible to improve the efficiency of the optimization search. In particular function monotonicity has been successfully exploited to improve statistical modeling, (e.g., Golchi et al., 2015) for analysis of computer experiments. The methods proposed here employ such monotonicity information for problems motivated by machine learning (ML), where the performance of an ML algorithm model is complex with respect its hyperparameters, which have to be tuned. We propose a sequential method that adapts the Bayesian optimization framework for an objective function that can be decomposed into a sum of functions with monotonicity constraints and exploit that structure. We analyze the method's applicability to ML hyperparameter problems and provide positive experimental results. Our algorithm incorporates monotonicity information in the Gaussian process (GP) model underlying Bayesian optimization. Bayesian optimization sequentially adds new objective function evaluations based on a probabilistic emulation of the objective trained using all current evaluations.