Sub-Architecture Ensemble Pruning in Neural Architecture Search

Yijun Bian 1, Qingquan Song 2, Mengnan Du 2, Jun Y ao 3, Huanhuan Chen 1, Xia Hu 2 1 University of Science and Technology of China 2 Department of Computer Science and Engineering, Texas A&M University 3 Data Science and Analytics Department, WeBank Abstract Neural architecture search (NAS) is gaining more and more attention in recent years due to its flexibility and the remarkable capability of reducing the burden of neural network design. To achieve better performance, however, the searching process usually costs massive computation, which might not be affordable to researchers and practitioners. While recent attempts have employed ensemble learning methods to mitigate the enormous computation, an essential characteristic of diversity in ensemble methods is missed out, causing more similar sub-architectures to be gathered and potential redundancy in the final ensemble architecture. To bridge this gap, we propose a pruning method for NAS ensembles, named as " Sub-Architecture Ensemble Pruning in Neural Architecture Search (SAEP) ." It targets to utilize diversity and achieve sub-ensemble architectures in a smaller size with comparable performance to the unpruned ensemble architectures. Three possible solutions are proposed to decide which sub-architectures should be pruned during the searching process. Experimental results demonstrate the effectiveness of the proposed method in largely reducing the size of ensemble architectures while maintaining the final performance. Moreover, distinct deeper architectures could be discovered if the searched sub-architectures are not diverse enough. Introduction Designing neural network architectures usually requires manually elaborated architecture engineering, extensive expertise as well as expensive costs. Neural architecture search (NAS), which aims at mitigating these challenges, is attracting more and more attention recently (Zoph et al. 2018; Elsken, Metzen, and Hutter 2019; Wistuba, Rawat, and Pedapati 2019).