A critical approach for efficiently deploying computationally demanding large language models (LLMs) is Key-V alue (KV) caching.

Large Language Models (LLMs) are widely used in today's tasks of natural language processing.

Code is available at github.com/spcl/QuaRot .

Key-value (KV) caching plays an essential role in accelerating decoding for transformer-based autoregressive large language models (LLMs).

Large language models(LLMs) have sparked a new wave of exciting AI applications. Hosting these models at scale requires significant memory resources.

Apart from the already-large model parameters, the key/value (KV) cache that holds information about previous tokens in a sequence can grow to be even larger than the model itself. This problem is exacerbated in one of the current LLM serving frameworks which reserves the maximum sequence length of memory for the KV cache to guarantee generating a complete sequence as they do not know the output sequence length. This restricts us to use a smaller batch size leading to lower GPU utilization and above all, lower throughput. We argue that designing a system with a priori knowledge of the output sequence can mitigate this problem.

To validate the robustness, we design extensive experiments across diverse prompts in terms of length, format, and content.