Robustifying Vision-Language Models via Dynamic Token Reweighting

Large vision-language models (VLMs) are highly vulnerable to jailbreak attacks that exploit visual-textual interactions to bypass safety guardrails. Rather than relying on curated safety-specific data or costly image-to-text conversion, we introduce a new formulation of the safety-relevant distributional shift induced by the visual modality. Large vision-language models (VLMs) (e.g., LLaVA Liu et al. (2023), InternVL Chen et al. (2024), and MiniGPT Zhu et al. (2024)) integrate vision and language capabilities, achieving remarkable multimodal modeling performance. However, incorporating visual modality introduces new vulnerabilities, making VLMs more susceptible to malicious manipulations than their backbone language models Liu et al. (2024). In multimodal jailbreaks, adversaries exploit the intricate interactions between visual and textual inputs to circumvent target VLMs' safety guardrails and elicit harmful responses Qi et al. (2023). A variety of attacks have been proposed, such as pairing harmful text with adversarially perturbed images Li et al. (2024), and embedding harmful content into images via generative models Liu et al. (2024) or typography Jiang et al. (2025).