Learning Dynamics of VLM Finetuning

The finetuning of vision-language models (VLMs) involves intricate learning dynamics that pose significant challenges for stable optimization (Liu et al., 2023; Huang & Zhang, 2024; Zhang et al., 2025a;c). VLMs process multimodal inputs, encoding textual and visual components as high-dimensional sequences, where the visual stream introduces complex state dependencies--such as pixel embeddings and spatial metadata--that tightly couple gradient updates across tokens (Radford et al., 2021; Li et al., 2023; Zhang et al., 2025d;b). Prominent finetuning methods, including supervised finetuning (SFT) (Ouyang et al., 2022) and direct preference optimization (DPO) (Rafailov et al., 2023), employ diverse loss geometries and supervision signals, necessitating a unified analytical framework to unravel their behavioral foundations, especially in preference-based alignment aimed at prioritizing human-preferred outputs (Ren & Sutherland, 2025). Preference-based finetuning is essential for aligning VLMs with human intent (Liu et al., 2024a; Radford et al., 2021; Chen et al., 2023; Zhang et al., 2024), yet it suffers from notorious instability in practice. Alignment datasets often contain static or mis-specified negative examples--trivially incorrect or off-distribution--that inject uninformative gradients (Casper et al., 2023; Kaufmann et al., 2024; Song et al., 2025). These gradients disrupt optimization, degrade calibration, and produce overconfident, peaky posteriors.