AttoNets: Compact and Efficient DNNs Realized via Human-Machine Collaboration

It is no secret that deep neural networks (DNNs) can achieve state-of-the-art performance in a wide range of complicated tasks. DNN models such as BigGAN, BERT, and GPT 2.0 have proved the high potential of deep learning. Deploying DNNs on mobile devices, consumer devices, drones and vehicles however remains a bottleneck for researchers. For such practical, on-device scenarios, DNNs must have a smaller footprint. The requirement for smaller DNNs has pushed researchers in two opposite directions: either hand-craft DNNs through design principles, or rely entirely on automated network architecture search.