ACHO: Adaptive Conformal Hyperparameter Optimization

Identifying optimal model parameters is deeply desirable for high prediction performance in machine learning, but challenging due to non-convexity and expensive search costs. Common approaches involving grid search - exhaustive iterative search of a confined parameter interval - or random search [1] - random sampling from a broader parameter space - display complimentary weaknesses and form no expectation of hyperparameter performance ahead of search. Focus has instead centered on search frameworks capable of forming hyperperameter performance expectations prior to sampling, generally dominated by Sequential Model-Based Optimization (SMBO) [2]. Early applications [2, 3] resulted in positive outperformance on expert consensus across a range of benchmarked datasets, leveraging Gaussian Process or Tree-structured Parzen estimators. Further expansions of the framework included search cost inclusion as an optimization criterion [4], early forms of online resource allocation and distributed search [5], unwanted parameter space pruning [6], or replacement of single estimators with ensemble methods [7].