High-Capacity Expert Binary Networks

Network binarization is a promising hardware-aware direction for creating efficient deep models. Despite its memory and computational advantages, reducing the accuracy gap between such models and their real-valued counterparts remains an unsolved challenging research problem. To this end, we make the following 3 contributions: (a) To increase model capacity, we propose Expert Binary Convolution, which, for the first time, tailors conditional computing to binary networks by learning to select one data-specific expert binary filter at a time conditioned on input features. Overall, our method improves upon prior work, with no increase in computational cost by 6%, reaching a groundbreaking 71% on ImageNet classification. A promising, hardware-aware, direction for designing efficient deep learning models case is that of network binarization, in which filter and activation values are restricted to two states only: 1 [36; 11]. This comes with two important advantages: (a) it compresses the weights by a factor of 32 via bit-packing, and (b) it replaces the computationally expensive multiply-add with bit-wise xnor and popcount operations, offering in practice a speedup of 58 on a CPU [36]. Despite this, how to reduce the accuracy gap between a binary model and its real-valued counterpart remains an open problem and it is currently the major impediment for their wide scale adoption. In this work, we propose to approach this challenging problem from 3 key perspectives: 1. Model capacity: To increase model capacity, we firstly introduce the first application of Conditional Computing [3; 2; 47] to the case of a binary networks, which we call Expert Binary Convolution. For each convolutional layer, rather than learning a weight tensor that is expected to generalize well across the entire input space, we learn a set of N experts each of which is tuned to specialize to portions of it.