Advancing Multimodal Large Language Models with Quantization-Aware Scale Learning for Efficient Adaptation

This paper presents the first study to explore the potential of parameter The remarkable performance of large language models (LLMs) has quantization for multimodal large language models to alleviate been well-established in recent literature [4, 9, 35, 36, 39], sparking the significant resource constraint encountered during visionlanguage a growing interest in the development of multimodal large language instruction tuning. We introduce a Quantization-aware models (MLLMs) [2, 3, 5, 24, 28, 32, 42]. This burgeoning field has Scale LeArning method based on multimodal Warmup, termed QS-led to substantial progress in a wide array of vision-language (VL) LAW. This method is grounded in two key innovations: (1) The tasks. To accomplish this, contemporary MLLMs primarily utilize learning of group-wise scale factors for quantized LLM weights multimodal instruction following examples for VL instruction tuning to mitigate the quantization error arising from activation outliers and adopt modular architectures [2, 21, 24, 28] to transform and achieve more effective vision-language instruction tuning; (2) visual features into the word embedding space of the LLM. This The implementation of a multimodal warmup that progressively innovative approach enables LLMs to execute multimodal tasks in integrates linguistic and multimodal training samples, thereby preventing an autoregressive fashion. One notable example of this technique is overfitting of the quantized model to multimodal data while LLaVA [24], which employs a linear projection layer to bridge the ensuring stable adaptation of multimodal large language models to gap between the visual encoder and the LLM.