Activating Associative Disease-Aware Vision Token Memory for LLM-Based X-ray Report Generation

Abstract--X-ray image based medical report generation achieves significant progress in recent years with the help of the large language model, however, these models have not fully exploited the effective information in visual image regions, resulting in reports that are linguistically sound but insufficient in describing key diseases. In this paper, we propose a novel associative memory-enhanced X-ray report generation model that effectively mimics the process of professional doctors writing medical reports. It considers both the mining of global and local visual information and associates historical report information to better complete the writing of the current report. Some researchers already exploit the effectiveness of LLM in the X-ray based medical report generation, such as R2Gen-GPT [1], R2Gen-I. This task can greatly alleviate the work pressure on high-quality text at the linguistic level, but they struggle to doctors and reduce the waiting time for patients, providing accurately identify abnormal conditions, diseases, and other a feasible method for empowering artificial intelligence in critical information in clinical diagnostic indicators. Although the task has made considerable result, although the obtained medical reports may appear to be progress in recent years, there are still many issues, such well-structured, they are actually difficult to address the practical as the difficulty in detecting key diseases and the challenge problems. In MRG, models typically need to process two shown in Figure 1, our framework contains two stages, i.e., the primary sources of information: visual information from medical disease-aware visual token mining and the associative memory images and linguistic information from existing medical augmented X-ray medical report generation. R2Gen [9] introduces a memory-driven the first stage, we extract the vision features of a given X-Transformer for radiology report generation, using relational ray image using the Swin Transformer network [4].