Foundations of Large Language Model Compression -- Part 1: Weight Quantization

In recent years, compression of large language models (LLMs) has emerged as an important problem to allow language model deployment on resource-constrained devices, reduce computational costs, and mitigate the environmental footprint of large-scale AI infrastructure. In this paper, we present the foundations of LLM quantization from a convex optimization perspective and propose a quantization method that builds on these foundations and outperforms previous methods. Our quantization framework, CVXQ, scales to models containing hundreds of billions of weight parameters and provides users with the flexibility to compress models to any specified model size, post-training. Large language Models (LLMs) have become a universal framework for solving a vast number of problems in natural language processing, from text translation and summarization to conversational AI and automatic generation of radiologist's reports. While LLMs outperform traditional methods in language-related tasks, they often involve tens or hundreds of billions of weight parameters, and this renders their deployment onto devices with limited resources challenging--model weights and activations no longer fit in the device memory, necessitating those activations be saved to and loaded from off-chip memory frequently during the course of inference.