Self-supervised vision-langage alignment of deep learning representations for bone X-rays analysis

In the medical domain, particularly in radiography, large-scale datasets are generally limited to English reports and to specific body areas. To the best of our knowledge, the only large publicly available radiography-report dataset is MIMIC-CXR[1], containing 377,110 Chest Xray images and their corresponding free-text reports in English. This raises a significant challenge when applying the models derived from those data to images other than Chest Xrays. Moreover, privacy regulations such as the General Data Protection Regulation (GDPR)[2] impose strict limitations on the distribution and sharing of medical databases containing sensitive patient information. To address this limitation, one viable approach would be to utilize local data available within a given hospital or healthcare institution. Hospitals typically maintain their own databases of medical images and associated reports, which are collected as part of routine clinical practice. While these local datasets may not be as extensive as publicly available datasets, they still contain valuable information that can be leveraged for training and evaluating machine learning models. Therefore, in this paper, we propose to explore vision-language pretraining using bone X-rays paired with French reports sourced from a single university hospital department. Specifically, our work aims at aligning deep embedding representations of Bone X-Rays and French Reports for solving image-based medical tasks with limited annotation.