fd78f2f65881c1c7ce47e26b040cf48f-Supplemental-Datasets_and_Benchmarks.pdf

License: Werelease the code used to build our benchmark and perform our experiments under theMITLicense (https://mit-license.org/),whereas werelease datawecreated, including the performance metrics collected by us, the splits used to train, validate and test our surrogate models, and our surrogate models, under the CCBY 4.0 License (https://creativecommons. Compute resources We trained the configurations on a large SLURM-based cluster with approximately 300,000 CPU-cores available in parallel. This ensures that all three data splits retain all or most of the statistical properties, including any biases, of the original performancedataset. Whereas fitting XGBoost used mean-squared-error as a regression metric, quality of fit for hyperparameters was judged using Kendall's tau rank correlation values. Task SpeedupoverHPO-only SpeedupoverNAS-only CIFAR-10 54.7 33.7 Colorectal-Histology 75.2 20.1 Fashion-MNIST 8.5 34.6 Geometricmean 32.7 28.6 resource consumption for our experiments performed on Intel(R) Xeon(R) Gold 6242 CPU @ 2.80GHztobe1.75CPU-core-hours.