Medical Large Vision-Language Models (Med-LVLMs) have been widely adopted for medical report generation. Despite Med-LVLMs producing state-of-the-art performance, they exhibit a bias toward predicting all findings as normal, leading to reports that overlook critical abnormalities. Furthermore, these models often fail to provide comprehensive descriptions of radiologically relevant regions necessary for accurate diagnosis. To address these challenges, we proposeMedical Report Generation Agents (MRGAgents), a novel multi-agent framework that fine-tunes specialized agents for different disease categories. By curating subsets of the IU X-ray and MIMIC-CXR datasets to train disease-specific agents, MRGAgents generates reports that more effectively balance normal and abnormal findings while ensuring a comprehensive description of clinically relevant regions. Our experiments demonstrate that MRGAgents outperformed the state-of-the-art, improving both report comprehensiveness and diagnostic utility.

Keywords: Visual - language model Biomedical artificial intelligence Model scaling Instruction tuning Multimodal learning ABSTRACT Objective T o advance b iomedical v ison - language m odel capabilities through scaling up, fine - tuning, and instruction tuning, develop vision - language models with improved performance in handling long text, explore strategies to efficiently adopt vision langu a ge mode l s for diverse multi - modal biomedical tasks, and examine the zero - shot learning performance. Methods We developed two biomedical vision language models, BiomedGPT - Large and BiomedGPT - XLarge, based on a n encoder - decoder - based transformer architecture. We fine - tuned the two models on 23 benchmark datasets from 6 multi - modal biomedical tasks including one image - only task (image classification), three language - only tasks (text understanding, text summarization and question answering), and two vision - language tasks (visual question answering and image captioning) . We compared the developed scaled models with our previous BiomedGPT - Base model and existi ng prestigious models reported in the literature . W e instruction - tuned the two models using a large - scale multi - modal biomedical instruction - tuning dataset and assessed the zero - shot learning performance and alignment accuracy . Results and Conclusion The experimental results show that the new models developed in this study outperform our previous BiomedGPT - Base model on 17 of 2 3 benchmark datasets and achiev e state - of - the - art performance on 15 of 23 datasets when compared to previous models reported in the literature . The new models also demonstrated improved ability in handling long text, particularly on text summarization on MIMIC - III dataset and text understanding on SEER dataset, with a remarkable improvement of 4.6~11.4 I nstruction tuning on the scaled models resulted in significant enhancements in zero - shot learning ability and alignment accuracy in following complex instructions across multiple tasks, including image classification, visual question answering, and image captioning . This study develop s two vision - language models in the biomedical domain and examine s technologies to improve long text content in vision language models through scaling, fine - tuning, and instruction tuning .

Medical Visual Question Answering (MVQA) systems can interpret medical images in response to natural language queries. However, linguistic variability in question phrasing often undermines the consistency of these systems. To address this challenge, we propose a Semantically Equivalent Question Augmentation (SEQA) framework, which leverages large language models (LLMs) to generate diverse yet semantically equivalent rephrasings of questions. Specifically, this approach enriches linguistic diversity while preserving semantic meaning. We further introduce an evaluation metric, Total Agreement Rate with Semantically Equivalent Input and Correct Answer (TAR-SC), which assesses a model's capability to generate consistent and correct responses to semantically equivalent linguistic variations. In addition, we also propose three other diversity metrics - average number of QA items per image (ANQI), average number of questions per image with the same answer (ANQA), and average number of open-ended questions per image with the same semantics (ANQS). Using the SEQA framework, we augmented the benchmarked MVQA public datasets of SLAKE, VQA-RAD, and PathVQA. As a result, all three datasets achieved significant improvements by incorporating more semantically equivalent questions: ANQI increased by an average of 86.1, ANQA by 85.1, and ANQS by 46. Subsequent experiments evaluate three MVQA models (M2I2, MUMC, and BiomedGPT) under both zero-shot and fine-tuning settings on the enhanced datasets. Experimental results in MVQA datasets show that fine-tuned models achieve an average accuracy improvement of 19.35%, while our proposed TAR-SC metric shows an average improvement of 11. 61%, indicating a substantial enhancement in model consistency.

Conventional task- and modality-specific artificial intelligence (AI) models are inflexible in real-world deployment and maintenance for biomedicine. At the same time, the growing availability of biomedical data, coupled with the advancements in modern multi-modal multi-task AI techniques, has paved the way for the emergence of generalist biomedical AI solutions. These solutions hold the potential to interpret different medical modalities and produce expressive outputs such as free-text reports or disease diagnosis. Here, we propose BiomedGPT, the first open-source and generalist visual language AI for diverse biomedical tasks. BiomedGPT achieved 16 state-of-the-art results across five clinically significant tasks on 26 datasets. Notably, it outperformed OpenAI's GPT-4 with vision (GPT-4V) in radiology human evaluation and surpassed Google's Med-PaLM M (12B) in breast cancer diagnosis and medical visual question answering. Moreover, BiomedGPT facilitates zero-shot transfer learning, greatly enhancing its utility as a biomedical assistant, similar to ChatGPT. Our method demonstrates effective training with diverse datasets can lead to more practical biomedical AI.