Spectrum: Targeted Training on Signal to Noise Ratio

Large language models (LLMs) have showcased impressive abilities across various natural language tasks. Efficiently training these massive models remains a major challenge, demanding extensive computational resources and time. To address this issue, a growing body of research has focused on developing techniques to minimize the memory footprint and accelerate LLM training without sacrificing performance. We present Spectrum, a method for selectively training the layers of an LLM based on their signal-tonoise ratio (SNR). Rooted in Random Matrix Theory, Spectrum utilizes the Marchenko-Pastur distribution to pinpoint informative layers according to their SNR. Unlike previous approaches, such as QLoRA [2], which quantizes the entire model, Spectrum strategically targets specific layers and modules for training while keeping others frozen. By concentrating computational resources on the most informative parameters, Spectrum achieves superior performance while significantly reducing training time and memory requirements compared to state-of-the-art methods. Our main contributions are as follows: We propose Spectrum, an approach for efficient LLM training that selectively trains layers based on their SNR.