Local Search is State of the Art for Neural Architecture Search Benchmarks

Neural architecture search (NAS) is a widely popular area of machine learning, with the goal of automating the development of the best neural network for a given dataset. Hundreds of NAS algorithms have been proposed [10, 40], and with the release of two NAS benchmark datasets [8, 39], the extreme computational cost for NAS is no longer a barrier, and it is easier to fairly compare different NAS algorithms. Most of the recently proposed state-of-the-art algorithms are becoming increasingly more complex, many of which use neural networks as subroutines [35, 36]. This trend is problematic because as the complexity of NAS algorithms increases, the amount of necessary "hyper-hyperparameter tuning", or tuning the NAS algorithm itself, increases. Not only is this a vicious cycle (will we start using AutoML algorithms to tune AutoML algorithms?), but the runtime for any hyper-hyperparameter tuning on a new dataset must be added to the total runtime of the NAS algorithm [19, 38]. Since this information is not always recorded, some NAS algorithms may have under-reported runtimes, making it harder to compare different algorithms.