Aggregated Learning: A Vector-Quantization Approach to Learning Neural Network Classifiers

We consider the problem of learning a neural network classifier. Under the information bottleneck (IB) principle, we associate with this classification problem a representation learning problem, which we call "IB learning". We show that IB learning is, in fact, equivalent to a special class of the quantization problem. The classical results in rate-distortion theory then suggest that IB learning can benefit from a "vector quantization" approach, namely, simultaneously learning the representations of multiple input objects. Such an approach assisted with some variational techniques, result in a novel learning framework, "Aggregated Learning", for classification with neural network models. In this framework, several objects are jointly classified by a single neural network. The effectiveness of this framework is verified through extensive experiments on standard image recognition and text classification tasks. Introduction The revival of neural networks in the paradigm of deep learning (LeCun, Bengio, and Hinton 2015) has stimulated intense interest in understanding the networking of deep neural networks, e.g., (Shwartz-Ziv and Tishby 2017; Zhang et al. 2017). Among various efforts, an information-theoretic approach, information bottleneck (IB) (Tishby, Pereira, and Bialek 1999) stands out as a fundamental tool to theorize the learning of deep neural networks (Shwartz-Ziv and Tishby 2017; Saxe et al. 2018; Dai et al. 2018). Under the IB principle, the core of learning a neural network classifier is to find a representation T of the input example X, that contains as much as possible the information about X and as little as possible the information about the label Y .