Iteratively Training Look-Up Tables for Network Quantization

Abstract--Operating deep neural networks (DNNs) on devices with limited resources requires the reduction of their memo ry as well as computational footprint. Popular reduction method s are network quantization or pruning, which either reduce the wo rd length of the network parameters or remove weights from the network if they are not needed. In this article we discuss a ge neral framework for network reduction which we call Look-Up T able Quantization (LUT -Q). For each layer, we learn a value dictionary and an assignment matrix to represent the network weights. W e propose a special solver which combines gradient descent an d a one-step k-means update to learn both the value dictionari es and assignment matrices iteratively. This method is very fle xible: by constraining the value dictionary, many different reduc tion problems such as nonuniform network quantization, traini ng of multiplierless networks, network pruning or simultaneo us quantization and pruning can be implemented without changi ng the solver . This flexibility of the LUT -Q method allows us to use the same method to train networks for different hardware capabilities. Deep neural networks (DNN)s are currently used in many machine learning and signal processing applications with g reat success as their performance often beats the previous state - of-the-art approaches by a large margin, e.g., see [2] for an overview of deep learning. DNN approaches have become standard practice in computer vision, automatic speech rec og-nition and partially in natural language processing. They a re also extensively investigated to support other domains lik e medicine, robotics and finance forecasting. Recently, there has been a lot of interest in the research community in reducing the memory/computational footprint of neural networks.