A resource-efficient method for repeated HPO and NAS problems

In this work we consider the problem of repeated hyperparameter and neural architecture search (HNAS).We propose an extension of Successive Halving that is able to leverage information gained in previous HNAS problems with the goal of saving computational resources. We empirically demonstrate that our solution is able to drastically decrease costs while maintaining accuracy and being robust to negative transfer. Our method is significantly simpler than competing transfer learning approaches, setting a new baseline for transfer learning in HNAS. Creating predictive models requires data scientists to delve into data sources, understand and visualize the raw data, apply multiple data transformations and pick a target metric. Searching deep learning architecture and optimization the hyperparameters are often left as a manual step to be performed "from time to time" in practice. However, best practice dictates that reusing historical architectures and hyperparameters under different experimental conditions can negatively impact the predictive performance.