Knowledge distillation has emerged as a powerful technique for compressing large language models (LLMs) into efficient, deployable architectures while preserving their advanced capabilities. Recent advances in low-rank knowledge distillation, particularly methods like Low-Rank Clone (LRC), have demonstrated remarkable empirical success, achieving comparable performance to full-parameter distillation with significantly reduced training data and computational overhead. However, the theoretical foundations underlying these methods remain poorly understood. In this paper, we establish a rigorous theoretical framework for low-rank knowledge distillation in language models. We prove that under mild assumptions, low-rank projection preserves the optimization dynamics, yielding explicit convergence rates of $O(1/\sqrt{T})$. We derive generalization bounds that characterize the fundamental trade-off between model compression and generalization capability, showing that the generalization error scales with the rank parameter as $O(r(m+n)/\sqrt{n})$. Furthermore, we provide an information-theoretic analysis of the activation cloning mechanism, revealing its role in maximizing the mutual information between the teacher's and student's intermediate representations. Our theoretical results offer principled guidelines for rank selection, mathematically suggesting an optimal rank $r^* = O(\sqrt{n})$ where $n$ is the sample size. Experimental validation on standard language modeling benchmarks confirms our theoretical predictions, demonstrating that the empirical convergence, rank scaling, and generalization behaviors align closely with our bounds.

Dubbing is a post-production process of re-recording actors' dialogues, which isextensively used infilmmaking and video production.

In this work, we propose Glow-TTS, a flow-based generativemodel for parallel TTS that does not require anyexternal aligner.

In this study, we introduce a novel model framework TransVIP that leverages diverse datasets in a cascade fashion yet facilitates end-to-end inference through joint probability.

As humans rely on audio localization cues for full scene awareness,spatial audio is a crucial componentof360 video.

In this work, we propose to use a bio-inspired architecture called Fully Recurrent Convolutional Neural Network(FRCNN) to solvethe separation task. This model containsbottom-up,top-downandlateral connections tofuse information processed atvarious time-scales represented by stages.

Oncetrained,ourmodelcangeneraterandom samples of arbitrary duration that maintain semantic similarity to the training waveform, yet exhibit new compositions of its audio primitives.

Our approach is based on a keyobservation about human speech: there isoften ashort pause between each sentence orword.

Sentence scoring aims at measuring the likelihood score of a sentence and is widely usedinnatural language processing scenarios, likereranking, which isto select the best sentence from multiple candidates.

Furthermore, it has also achievedstate-of-the-art performance incombination with recent developments inself-supervised learning methodologies as well [37,62].