Efficient Automation of Neural Network Design: A Survey on Differentiable Neural Architecture Search

The automation of this field supported the development of novel Deep Learning (DL) [62] architectures, especially Convolutional Neural Networks (CNN) [61], that competed with previous state-of-the-art handcrafted models. Since the introduction of CNNs with LeNet [61] and the beginning of Deep Learning with AlexNet [60], most improvements in the field (e.g., deepening the architecture or adding residual connections) were driven by empiricism. NAS aims to put an end to this trial-and-error practice and bring a formal way to smoothen the progress in deep learning architecture design. Moreover, automatically discovering more efficient architectures is particularly relevant in the ecological transition context (i.e., green Deep Learning [119]). The role of manual feature engineering and model development has gradually decreased ever since.