Increasing Textual Context Size Boosts Medical Image-Text Matching

Pretrained image-text matching models, such as OpenAI's CLIP [1], use natural language processing (NLP) approaches to find semantic relations between images and textual descriptions. This emerging technology has seen rapid adoption in the general domain, and increasing interest in the medical domain [2, 3] where medical imaging data often includes images paired with textual descriptions. For example, MIMIC-CXR[4] is a dataset that consists of chest radiographs along with free-text radiology reports. This dataset paved the way for works like BioViL [2] which used the images and the captions provided in the dataset to train an image-text matching model for chest X-Rays and chest related diseases. ROCO [5] is a dataset containing radiology images from publications available in the PubMed biomedical paper repository. ROCO includes several medical imaging modalities beyond X-Ray, such as CT, Ultrasound and MRI.