Large language models (LLMs) with long-context capability, such as GPT-4 [1], Gemini [42], and LWM [24] continue to emerge and gain proficient application in scenarios including chatbots, vision generation, and financial analysis [9, 35, 43, 54]. However, losslessly serving these LLMs efficiently is challenging. Because of the auto-regressive nature of LLMs, the entire key-value (KV) cache, which stores intermediate key-value states from previous contexts to avoid re-computation, together with model parameters will be loaded into GPU SRAM for every token generated, resulting in low utilization of computational cores. In addition to the large volume of model parameters, the memory footprint of KV cache, which grows linearly with sequence length [33], is emerging as a new bottleneck for long sequence generation. Recent methodologies have proposed KV cache eviction strategies [13, 17, 28, 47, 53] to mitigate the substantial memory footprint of KV cache, which selectively discard KV pairs from the cache based on a designed eviction policy, allowing models to generate texts with a limited KV cache budget. However, considering that discarded KV pairs cannot be restored and the difficulty in precisely foreseeing which KV pairs will be crucial for future text generation, they struggle with potential information loss, including hallucination and contextual incoherency [49], particularly in long contexts. Such challenges prevent these approaches from boosting speed without sacrificing the performance of models, as illustrated in Figure 1. Concurrently, speculative decoding, which leverages a lightweight draft model to sequentially predict the next few tokens and let the target model verify the predicted tokens in parallel, is introduced to accelerate LLM inference while provably precisely preserving model output [6, 22, 45].