Dynamic Encoder Size Based on Data-Driven Layer-wise Pruning for Speech Recognition

In this work, we combine the benefits of both ideas and demonstrate an efficient dynamic encoder training framework. Varying-size models are often required to deploy ASR systems We leverage score-based layer-wise pruning to find the optimal under different hardware and/or application constraints such layer combination for the subnets, saving the computationally as memory and latency. To avoid redundant training and optimization expensive search required by the general supernet training efforts for individual models of different sizes, we methods [9, 10]. Furthermore, we design an efficient two-step present the dynamic encoder size approach, which jointly trains training pipeline. In Step 1, we propose two methods, Simple-multiple performant models within one supernet from scratch. Top-k and Iterative-Zero-Out, to effectively learn the associated These subnets of various sizes are layer-wise pruned from the layer importance scores in a data-driven way. In step 2, we generate supernet, and thus, enjoy full parameter sharing. By combining binary masks for all subnets and exploit the sandwich rule score-based pruning with supernet training, we propose two [6] for efficient joint training of the supernet and subnets. Additionally, novel methods, Simple-Top-k and Iterative-Zero-Out, to automatically we explore different training techniques to mitigate select the best-performing subnets in a data-driven the mutual training inference and further boost the word error manner, avoiding resource-intensive search efforts.