SpecVLM: Fast Speculative Decoding in Vision-Language Models

Speculative decoding is a powerful way to accelerate autoregressive large language models (LLMs), but directly porting it to vision-language models (VLMs) faces unique systems constraints: the prefill stage is dominated by visual tokens whose count scales with image resolution and video length, inflating both compute and memory--especially the key-value (KV) cache. We study speculative decoding for VLMs and introduce SpecVLM, a practical system that (1) establishes a strong EAGLE-2-style baseline, EagleVLM, delivering 1.5-2.3 To avoid costly offline distillation corpora, we propose an online-logit distillation protocol that trains the draft model with on-the-fly teacher logits and penultimate features using a combined cross-entropy and Smooth L1 objective, eliminating storage and preprocessing while remaining compute-efficient. This protocol reveals a training-time scaling effect: longer online training monotonically increases the draft model's average accepted length, improving speculative efficiency. Empirically, SpecVLM achieves additional acceleration, culminating in 2.5-2.9 end-to-end speedups within 5 epochs across LLaV A and MMMU, consistently over resolutions and task difficulties, while preserving the target model's output distribution (lossless decoding). Autoregressive decoding underpins many high-quality vision-language models (VLMs) such as LLaV A (Liu et al., 2023), GPT -4 (Achiam et al., 2023), and Gemini (Team et al., 2023), which are widely used for image captioning, visual question answering, and multimodal dialogue. While these teacher models produce high-fidelity outputs, their token-by-token decoding is computationally expensive--an issue that is amplified in multimodal settings because the prefill stage (visual encoding projection token injection) can dominate wall-clock time and memory usage (Li et al., 2025b). Higher image resolutions, denser visual tokenizations, and video inputs dramatically increase the number of visual tokens, which in turn inflates both the prefill cost and the KV cache traffic during decoding.