Energy-Aware Neural Architecture Optimization with Fast Splitting Steepest Descent

A BSTRACT Designing energy-efficient networks is of critical importance for enabling state-of-the-art deep learning in mobile and edge settings where the computation and energy budgets are highly limited. Recently, Liu et al. (2019b) framed the search of efficient neural architectures into a continuous splitting process: it iteratively splits existing neurons into multiple offsprings to achieve progressive loss minimization, thus finding novel architectures by gradually growing the neural network. However, this method was not specifically tailored for designing energy-efficient networks, and is computationally expensive on large-scale benchmarks. In this work, we substantially improve Liu et al. (2019b) in two significant ways: 1) we incorporate the energy cost of splitting different neurons to better guide the splitting process, thereby discovering more energy-efficient network architectures; 2) we substantially speed up the splitting process of Liu et al. (2019b), which requires expensive eigen-decomposition, by proposing a highly scalable Rayleigh-quotient stochastic gradient algorithm. Our fast algorithm allows us to reduce the computational cost of splitting to the same level of typical back-propagation updates and enables efficient implementation on GPU. Extensive empirical results show that our method can train highly accurate and energy-efficient networks on challenging datasets such as ImageNet, improving a variety of baselines, including the pruning-based methods and expert-designed architectures. 1 I NTRODUCTION Deep neural networks (DNNs) have demonstrated remarkable performance in solving various challenge problems such as image classification (e.g. Simonyan & Zisserman, 2015; He et al., 2016; Huang et al., 2017), object detection (e.g. Although large-scale deep networks have good empirical performance, their large sizes cause slow computation and high energy cost in the inference phase.