Scaling Up Biomedical Vision-Language Models: Fine-Tuning, Instruction Tuning, and Multi-Modal Learning

Keywords: Visual - language model Biomedical artificial intelligence Model scaling Instruction tuning Multimodal learning ABSTRACT Objective T o advance b iomedical v ison - language m odel capabilities through scaling up, fine - tuning, and instruction tuning, develop vision - language models with improved performance in handling long text, explore strategies to efficiently adopt vision langu a ge mode l s for diverse multi - modal biomedical tasks, and examine the zero - shot learning performance. Methods We developed two biomedical vision language models, BiomedGPT - Large and BiomedGPT - XLarge, based on a n encoder - decoder - based transformer architecture. We fine - tuned the two models on 23 benchmark datasets from 6 multi - modal biomedical tasks including one image - only task (image classification), three language - only tasks (text understanding, text summarization and question answering), and two vision - language tasks (visual question answering and image captioning) . We compared the developed scaled models with our previous BiomedGPT - Base model and existi ng prestigious models reported in the literature . W e instruction - tuned the two models using a large - scale multi - modal biomedical instruction - tuning dataset and assessed the zero - shot learning performance and alignment accuracy . Results and Conclusion The experimental results show that the new models developed in this study outperform our previous BiomedGPT - Base model on 17 of 2 3 benchmark datasets and achiev e state - of - the - art performance on 15 of 23 datasets when compared to previous models reported in the literature . The new models also demonstrated improved ability in handling long text, particularly on text summarization on MIMIC - III dataset and text understanding on SEER dataset, with a remarkable improvement of 4.6~11.4 I nstruction tuning on the scaled models resulted in significant enhancements in zero - shot learning ability and alignment accuracy in following complex instructions across multiple tasks, including image classification, visual question answering, and image captioning . This study develop s two vision - language models in the biomedical domain and examine s technologies to improve long text content in vision language models through scaling, fine - tuning, and instruction tuning .