Is Differentiable Architecture Search truly a One-Shot Method?

Recent progress in computer vision and related fields has illustrated the importance of suitable neural architecture designs and training schemes He et al. [2015]. Ever deeper and more complex networks show promise, and manual network design is less and less able to explore the desired search spaces. Neural architecture search (NAS) is the task of optimizing the architecture of a neural network automatically without resorting to human selection, scaling to larger search spaces and proposing novel well-performing architectures. NAS, which is an intrinsically discrete problem, has been successfully addressed using black-box optimization approaches such as reinforcement learning Zoph and Le [2017], Zoph et al. [2018] or Bayesian optimization Kandasamy et al. [2018], White et al. [2019], Ru et al. [2020], Lukasik et al. [2021]. However, these approaches are computationally expensive as they require the training of many candidate networks to cover the search space. In contrast, differentiable architecture search (DAS) Liu et al. [2019] proposes a continuous relaxation of the search problem, i.e. all candidate architectures within a given search space of operations and their connectivity are jointly optimized using shared network parameters while the network also learns to weigh these operations. The final architecture can then simply be deduced by selecting the highest weighted operations. This is appealing as practically good architectures are proposed within a single optimization run. However, previous works such as Zela et al. [2020] also indicate that the proposed results are often sub-optimal, especially when the search space is not well chosen.