Progressive Label Distillation: Learning Input-Efficient Deep Neural Networks

Deep learning has been widely adapted to many different problems, such as image classification [1], speech recognition [2] and natural language processing [3], and has demonstrated state-of-the-art results for these problems. Despite the promises, deep neural networks (DNNs) remain challenging to deploy in on-device edge scenarios such as mobile and other consumer devices. Due to the limited computational resources available in such on-device edge scenarios, many recent studies [4, 5, 6, 7] have put greater efforts into designing small, low-footprint deep neural network architectures that are more appropriate for embedded devices. A particularly interesting approach for enabling low-footprint deep neural network architectures is the concept of knowledge distillation [8], where the performance of a smaller network is significantly improved by leveraging a teacher-student strategy where the smaller network is trained to mimic the behaviour of a larger teacher network. With much of the research around distillation focused on distilling knowledge from larger networks to smaller networks, there is little research focused on leveraging the concept of distillation for distilling knowledge encapsulated in the training data itself into a reduced form. By producing data with reduced data dimension, one can achieve input-efficient deep neural networks with significantly reduced computational costs. In this study, we explore a concept we will call progressive label distillation, where a series of teacher-student network pairs are leveraged to progressively generate distilled training data.