Efficient Prompt Compression with Evaluator Heads for Long-Context Transformer Inference

Large language models (LLMs) have exhibited exceptional capabilities in a variety of real-world tasks and applications, with an increasing need for processing long inputs in areas such as literary novels, legal documents, instruction manuals, and code documentation. Inference tasks that requires understanding of long contexts, such as long document summarization (Zhang et al., 2024), reasoning (Fei et al., 2024a), and autonomous agents (Singh et al., 2024; Chen et al., 2024), are of particular importance due to the high stakes in these scenarios. However, the deployment of LLMs is challenged by the computational and memory demands inherent to transformer-based architectures, resulting in increased latency, particularly when processing lengthy input prompts. Prompt compression, which entails substituting the input prompts provided to a language Table 1: Overall comparison of the proposed model with more succinct versions, has surfaced method in terms of average performance and latency as a promising strategy for enhancing long-text on the LongBench dataset, under the constraint understanding and mitigating associated costs. of a compressed prompt length of 2048 tokens. Current mainstream methods, such as Select-For comprehensive results, please see Table 4 Context (Li et al., 2023), LLMLingua (Jiang and Table 5. et al., 2023a) and LongLLMLingua (Jiang et al., 2023b), typically rely on pre-trained LLMs, utilizing the logits or perplexity of the prompts